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FOREWORD 



When the U.S. Office of Educa- 
tion was chartered in 1867* 
one charge to its commission- 
ers was to determine the na- 
tion's progress, in education. 
The National Assessment of 
Educational Progress (NAEP) 
was initiated a century latfer 
to address, in a systematic 
way, that charge. 

Since 1969, the National As- 
sessment has gathered informa- 
tion about levels of educa- 
tional achievement across the 
country and reported its find- 
ings to the nation. It has 
surveyed the attainments of 
9-year-olds, 13-year-olds , 

17-year-olds^ and sometimes 
adults in art, career and oc- 
cupational development, citi- 
zenship, literature, mathemat- 
ics, music, reading, science, 
social studies and writing. 
All areas have been periodi- 
cally reassessed in order to 
detect any important changes. 
Including students who partic- 
ipated in the latest art as- 
sessment, National Assessment 
has interviewed and tested 
more than 1,000,000 young 
Americans. 

Learning-^rea assessments 
evolve* from a consensus proc- 
ess. Each assessment is the 
product of several years of 
work by a great many educa- 
tors, scholars and lay persons 
from all over the nation. 
Initially, thr people design 



objectives for each subject 
area, proposing general goals 
they feel -Americans should be 
achieving in the course of 
* their education. After careful 
reviews, these objectives are 
given to exercise (item) writ- 
ers, whose task it is to cre- 
ate measurement instruments 
appropriate to the objectives. 

When the exercises have- passed, 
extensive reviews by subject- 
matter specialists, measure- 
ment experts and lay persons, 
thny are administered to prob- 
ability samples. The peoples 
who compose these samples are 
■ chosen in such a way that the 
results of their assessment 
can be generalized to an en- 
tire national population. 
That is, on the basis of the 
performance of about 2,500 
9-year-olds on a given exer- 
cise, we can make generaliza- 
tions about the probable per- 
formance of all 9-year-olds in 
the nation. 

After assessment data have 
been col lecteST - scored and 
analyzed, the National Assess- 
ment publishes reports and 
disseminates the results as 
widely as possible. Not all 
exercises are released for 
publication. Because NAEP will 
readminister some of the same 
exercises- in the future to 
determine whether the perform- 
ance levels of Americans have 
increased, remained stable or 



vii 



decreased, it is essential - order to preserve the. integ- 
that they not be released in rity of the study. . 
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The National Assessment of 
Educational 'Progress w (NAEP) 
has completed two assessments 
of art, the first conducted in 
1974-75 and *the second in 
1978-79. Each assessment sur- 
veyed the achievement and at- 
titudes of American 9-, 13- 
and ' 17-yearr-olds, using a 
deeply stratified, multistage v 
probability ' sample design. 
This report documents proce- 
dures used in the 1378-79 art 
assessment and also describes 
change's in procedures between 
the assessments * , 

To measure ^changes in perform- 
ance between 1974-75 and 
1978-79, approximately half of 
tHe exercises assessed in, the 
first assessment were reas- 
sessed in the second under 

>almost identical 'administra- 
tive conditions. To measure 
the status of art' achievement 

* in 1978-79, National Assess- 
ment consultants reviewed the 
objectives used in the first 
assessment and developed addi- 
tional exercises to provide 
wider coverage of those objec- 
tives. 1 



Althpugh National Assessment 
traditionally < revises objec- 
tives prior ,to an assessment, 
a shortage of funds and time 
prohibited complete redevelop- 
ment of the art objectives. 
Previous art objectives were 



Approximately 7, 500 9-year-' 
bids, ll f 000 13-year-olds and 
13,500 17-year-olds partici- 
pated in the 1978-79 art as- 
sessment. Because * there were 
more art exercises for 13- 
year-olds than available as-? 
sessment space , six exercises 
were -held and administered in 
the next year's assessment 
(1979-80). During the 1979-80 
assessment, 2 r 749 13-year-olds 
responded to these six art 
exercises. , 

Since National Assessment re- 
ports results for groups of 
students, not individuals, it 
is not necessary for each stu- 
dent to respond to every item 
(exercise). 2 Each respondent 
completed only one item book- 
let^of about 45 minutes in' 
lefigth. Between 2,400 and - 
2,800 students" responded to 
each booklet. In 1978-79, 
three exercise booklets for 
9-year-olds, four booklets for 
" 13-year-olds and five booklets 



reviewed by consultants and 
found \o be usable with some 
shifts in x empha"sis . . 

2 National Assessment uses the 
term "exercise 1 **" ,to mean an 
assessment item. The terms 
"exercise" and "item" are used 
interchangeably in this re- 
port. 
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for 17-year-olds contained art 
exercises. , In the 1979-80 as- 
sessment, only one item book- 
let for 13-year-olds included 
art items. 

O t 

In both the '1974-75 and the 
1978-79 assessments, 13-year- 
olds were assessed in October 
through December ,~ 9-year-olds^ 
in January r,nd February ancT 
17-year-olds in March and 
Apri.1. Thus, the amount of 
school experience for each age 
group was approximately the 
same in each assessment. 

The exercises for each assess- 
ment were administered by a 
professional data collection 
staff to minimize the burden 
on participating schools and 
to maximize the uniformity of 
assessment conditions. In- 
structions and items were re- 
corded on a paced audio tape 
and played back to students to 
reduce the potential effect of 
reading difficulties and to , 
ensure -that all students moved 
through the packages* at the 
same speed. 

The majority of the ar£ items 
were multiple-choice ; a few 
exercises from the first as- 
sessment that were reassessed 
in 1978-79 were open-ended. 
Many items included more than 
one part. 

Multiple-choice items were 
scored by an optical scanning 
machine; open-ended items were 
hand-scored by trained scorers 
using scoring guides developed * 
to define categories of 
acceptable and unacceptable 
responses. 

* 

National Assessment reports 



estimated percentages of* cor- 
rect responses for single 
items. When a report indicates 
that "85% of the 13 7 year-plds 
gave a correct response , H it 
means that an estimated J85% of 
the 13-year-olds would" have 
given a correct response if 
all % the 13-year-olds in 
schools across the country had 
been assessed. National As- 
sessment also aggregates per- 
centages of success on various 
sets of items to provide data 
on changes in performance be- 
tween assessments and on the 
differential performance of 
population subgroups . In addi- 
tion to reporting national 
results , National Assessment 
provides 'data on the perform- * 
ance of various population 
subgroups within the national 
population, defined by sex, 
race/ethnicity, region of the^ 
country, size and type of com- 
munity lived in and level of 
parental education. In addi- 
tion, for the art assessments, 
some data are available on the * 
extent of art training and/or 
experience with art — either 
fojrmal or informal — that 9-, . 
13- and 17-year-olds have had. 



This handbook describes the 
procedures used to develop, 
administer and analyze the 
results of the 1978-79 art 
assessment. The primary pur- 
pose of this handbook is to 
provide d eta i led procedural 
information for people 

interested in replicating the 
assessment or in need of more 
information than is provide^ 
in the reports containing as- 
sessment data. The seven chap- 
ters cover; objectives redevel- 
opment, exercise creation , 
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preparation of assessment 
booklets, sampling, data col- 
lection,; scoring and data 
analysis. Each chapter ex- 
plains' the basic procedures 
used for the 1978-79 art as- 
sessment and contrasts the 
procedures to those used in 
earlier* years (if »there were 
changes).. 

Appendixes include definitions 
of reporting groups used by 
National Assessment, forms 
used to gather background in- 
formation about students and 
schools, response' rates, com- 
putation of achievement meas- 
ures and procedures for 
smoothing respondent weights. 
A glossary of National Assess- 
ment terms is provided* at the 
eWd of the book. 

The objectives for the art 
assessments, Art Objectives 
(1971) and a one page objec- 



tives supplement and released 
exercises, * The Second Assess- 
ment of Art, 1978-79; Released 
Exercise > Set ~ ( 1980) .,\ / ar£ t 
available from NAEP. 



A supplement to the released 
exercise set that will include 
released open-ended exercises , 
their scoring guides and some 
selected data is planned fo r 
publication in late 1981. By 
the end of 1981, National As- 
sessment also will complete 
reports on the results of the 
1978-79 art assessment and the 
changes in art performance 
between the 1974-75 and 1978- 
79 assessments. For the exact 
ti ties of these publications 
and 'information about " their 
availability, more complete 
data or other assistance con- 
cerning the art assessments, 
please contact National As- 
sessment. 
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CHAPTER 1 
OBJECTIVES REDEVELOPMENT 



The primary goals of the Na- 
tional Assessment of Educa- 
tional Progress (NAEP) are to 
report on th«_current educa- 
tion status of young Americans 
and to monitor any changes in 
achievement over time. . For 
each learning, area to be as- 
sessed, NAEP asks consultants 
to develop objectives that 
define the subject area. Since 
the objectives provide guide- 
lines for exercise developers, 
consultants , are ar'~ed to in- 
clude examples of the knowl- 
edge, skills and attitudes to 
be assessed at e°ach age level. 

Education in America is a col- 
laborative enterprise involv- 
ing a great many people with - 
wideiy^if fering philosophies. 
Providing information about 
education nationwide would be 
considerably easier if there 
were a consensus about the 
means and ends of American 
education, hut the fact is 
that Americans" have conflict- 
ing' and sometimes contradic- 
tory values regarding the^ * 
goals of education and^ , the 
means for achieving them'i To 
develop an assessment that. is 
truly national in scope and 
takes into account the diver- 
sity of curricula, values and 
goals across the country, Na- 
tional Assessment employs a 
consensus process for develop- 
ing objectives, with represen- 



tation of many different 
groups of people. 

Several types of consultants 
helped develop the objectives 
for the first ass sment of 
art. College and university 
specialists in art, classroom 
teachers, curriculum supervi- 
sors and persons involved in 
teacher education made " sTTFe- 
that the objectives included 
important concepts, skills and 
attitudes that the, schools 
currently were teaching and 
that *they should be teaching. 
Concerned citizens , parents 
and other interested lay per- 
sons also had to agree that 
the objectives were important 
for young people * to achieve, 
were free of education jargon 
and were .not biased against or 
offensive to any groups. Con- 
sultants were selected to rep- 
resent different regions of 
the country and minority 
groups. They also represented 
*a range, of experience C/with 
students of different ages, and 
community types. 

In preparation for the 1978-79 
art assessment, the 1974-75 
art objectives were reviewed* 
by art educators. A shortage 
of funds and" a very tight 
schedule for completing the 
assessment made a complete 
redevelopment of the objec- 
tives impractical. To reflect 
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shifts of emphasis in art edu- 
cation, the objectives were 
reviewed and the amount- of 
assessment time to be devoted 
to each objective was modified 
as necessary. Suggestions for 
more extensive revisipns were 
provided to Nat ional Assess- 
ment for future implementa- 
tion. 

As seen in Tables 1 and 2, for 
the 1978-79 art assessment, 
the objectives were, assigned 
the same percentage of time, 
or weight, for each age, al- 
though \ the weights used in 
1974-75 varied .with age for 
some objectives. The major 
shift between assessments was 



that the production objective 
(III) received more ejnphasis 
for 13- and 17-year-olds, with 
the amount of time for objec- 
tives concerning knowledge 
about art; (IV) and aesthetic 
judgments (V / correspondingly 
reduced. 

Although in 1974-75 percent- 
ages of time were assigned 
only to ma jor objectives, in, 
1978-79 percentages were as- 
signed to subobjectives as 
well, to give additional gui- 
dance to exercise developers. 

An outline of the art objec- 
tives used ii» 1974-75 and 
1978-79 1 follows. For addi- 



TABLE 1. Desired Percent of Assessment Exercise Time 
by Objective and Subobjective, for Ages 9, 13 and 17, 

1978-79 Assessment 









Objective 






Subobjective 


X 


II 


III 


IV 


V 


A 


8.0% 


8.0% 


6.0% 


3.0% 


3.0% 


B f 


12.0 


4.8 


6.0 


3.0 


3.0 


C 


NA+ 


2.4 


6.0* 


3.0 


3.0 


D 


NA . "» 


2.4 


6.0 


3.0 


6.0 


E 


NA 


2.4 


6.0 


, 3.0 


NA 


Total 


20. 0-- 


20.0 


30.0 


15.0 


15.0 


+Not assessed. 












TABLE 2, Desired 


Percent 


of Assessment ; 


xercise 


Time 


by Age and Major Objective, 


1974-75 Assessment 


» 


*I • 




Objective 








II • • 


III 


IV 


V 


Age 9 ' 


2Q.0% 


20.0% 


30.0% 


15.0% 


15.0% 


Age. 13 


20.0 


20.0 


25.0 


17.5 


17.5 


Age 17 


20.0 


20.0 


20.0 


20.0 


20. 0 
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tional detail concerning art 
objectives, see Art Objectives 
(1971) . 



Outline of Art 
Objectives . 

I. PERCEIVE AND RESPOND TO 
ASPECTS OF ART 

Aspects of art are defined as: 
sensory qualities of color, 
line, shape and texture; com- 
positional elements such as 
structure, space, design, bal- 
ance, movement,. placement, 
closure, contrast and pattern; 
expressive qualities "such as 
mood, feeling and emotion; 
subject matter, including (1) 
objects, themes (the general 
subject of a work, i.e., land- 
scape or battle scene) , events 
and ideas (general presymbolic 
meanings) and (2) symbols and 
allogories; and expressive 
content, which is a unique 
fusion of the foregoing as- 
pects 

A. Recognize, and describe 
the subject-matter 
elements of .works of 

- art 

B. Go beyond the recogni- 
tion of subject matter 
jto the perception and 
descript-. on of formal 

. qualities and expres- 
sive content (the com- 
bined effect of the 
subject matter and the 
specific visual form 
that characterizes a 
particular work of 
art) 
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II. VALUE ART AS AN IMPORTANT 
REALM OF HUMAN EXPERIENCE 

A. Be affectively ori- 
ented toward art 

B. Participate in activi- 
ties related to art 

C. Express reasonably 
sophisticated concep- 
tions about and posi- 
tive attitudes toward 
art and artists 

D. Demonstrate an open- 
mindedness toward dif- 
ferent forms and 
styles of art 

E. Demonstrate an open- 
mi ndedness toward ar- 
tistic experimentation 



III. PRODUCE WORKS OF ART 

A. Produce original and 
imaginative works of 
art " 

B. Express visual ideas 
fluently 

C. Produce works of art 
with a particular com- 
position, subject mat- 
ter, expressive char- 
acter pr expressive 
content 

*D. Produce works of art 
that contain various 
visual conceptions 

E. Demonstrate knowledge 
and application of 
media, tools, techni- 
ques and forming pro- 
cesses' 
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IV. KNOW ABOUT ART 

A. Recognize major fig- 
ures*^ and^works in the 
history of art and 
understand their sig- 
nificance. (Signif- 
icance as it is used 
here refers to such 
things as works of art 
that began new styl.es , 
markedly influenced 
subsequent wo rks / 
changed the direction 
of art*, contained vis- 
ual and technical dis- 
cove r ies , expressed 
particularly well the 
spirit of their age 
and those considered 
to be the major works 
of major artists) 

B. Recognize the styles 
of art r understand the 
concept of style and 
analyze works of art 

- on the__bas i s of s tyl e 

C. Know the history of 
art activity and 
understand the* rela- 
tion of one style or 
period to other styles 
and periods 

D. Distinguish ' ^ between 
factors of a " work of 
art that^ relate, prin- 
cipally to the per- 
sonal style " of the 
artist and factors 
that relate , to the 
stylistic * period or 
the entire age 

E. Know and recognize the 
* . relationships that 



existed between art 
and the other disci- 
plines of the humani- 
ties (literature, 
music and particularly 
the history of ideas 
and philosophy) during 
a given period 



V. MAKE AND JUSTIFY 

JUDGMENTS ABOUT THE AES- 
THETIC MERIT AND QUALJTY 
OP WORKS OP ART 

Statements of aesthetic qual- 
ity are those that character- 
ize the various aspects of a 
work of art, while statements 
of aesthetic merit are asser- 
tions about th6 degree of 
^goodness or badness of the 
work. Justifications of aes- 
thetic merit are based on cri- 
teria such as the degree to 
which . the work is integrated 
and whether contact with the 
work results in a vivid and 
fused experience 

A. Make and justify judg- 
ments about aesthetic 
me r i t 

B. Make and justify judg- 
ments about aesthetic 
quality 

C. Apply specific crite- 
ria in judging works 
of -art 

D. Know and understand 
criteria ' for making 
aesthetic judgments 
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CHAPTER 2 



DEVELOPMENT 



Exercise Development 

* 

Exercise development for the 
1978-79 art assessment began 
with a planning conference 
early in 1977. At this con- 
ference, consultants evaluated 
exercises developed for the 
first assessment but not as- 
sessed and determined which 
objectives tfere not adequately 
measure^ by existing -materi- 
als. In addition, prototype 
items that could be used as 
models by exercise writers , 
were identified . 

Consultants who are experts in 
-art, child development and art 
education wrote new exercises 
during March 1977. The new 
exercises and those chosen 
from the items developed for 
the first assessment but not 
used were field tested in 
spring 1977. The exercises and 
the results of the field 
trials were reviewed by 
consultants during summer 
1977. This review provided 
guidance for further exercise 
development, which was 
conducted in late summer and 
early fall 1977. Exercises 
from this development were 
field tested during October 
and November. Fallowing ,these 
field tests,, a 3,1 newly 
developed exercises and field 
test results were reviewed by 



OF EXERCISES 

subject-matter experts and lay 
persons . 

Subject-matter .specialists 
reviewed exercises to ensure 
that they were appropriate for 
the age level being assessed 
and that the content included 
was correct. Lay citizens, 
^representing a variety of oc- 
cupations and interests, also 
reviewed the exercises, check- 
ing for sex or racial/ethnic 
bias and considering the gen- 

* eral importance of each exer- 

* cise. % 

By the end of February 1978, 
exercises had been revised and 
edited and were ir^Jinal form 
for inclusion in the assess- 
ment. Since more t items were 
developed than could be as- 
sessed, in March the items to 
be used 5or the first time in 
the 1978-79 . assessment were 
selected. Exercises to be used 
were selected at a conference 
of subject-matter experts and 
educators. Items from the 
first assessment to be .used to 
measure changes in performance 
were automatically^ included in 
the second assessment. Using 
the weighted objectives' as a 
" guide to the amount of assess- 
ment time to be used for each 
subobiective, exercises (in- 
cluding those from the first 
assessment) were selected to 
approximate these amdunts. 



Field Tryout Procedures 



The field trials ^for the 
1978-79 art assessment were 
performed in' schools across 
the co'untry to discover poten- 
tial problems ^with items in 
wording , directions or admin- 
istrative procedures and to 
collect item statistics, tim- 
ing information and scoring 
information. "Tryout" schools 
were selected to represent 
high- and low-income communi- 
ties as well as more typical 
communities. The items being 
field tested were administered 
to students in at least four 
classrooms (approximately 100 



students) 
which an 
In order 
assessment 



at each age for 
item was intended, 
to simulate actual 

field procedures , 



students recorded their an- 
swers in the test booklets; 
directions and questions were 
read to studeats from an audio 
tape; and National Assessment 
staff members , rather than 
class roc v teachers, adminis- 
tered the test. The comoleted 
"tryout" packages were then 
scored anu results were ana- 
lyzed. The tryout data, as 
well as the administrators 1 
reports^of "any field problems, 
helped "'both NAEP staff and 
consultants to , evaluate and 
revise the exercises . 
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CHAPTER 3 
PREPARATION OF ASSESSMENT MATERIALS 



Preparation of Booklets 
and Audio Tapes 

Na-tional Assessment uses a 
matrix sampling approach, with 
different nationally represen- 
tative samples of students 
responding to different item 
booklets (see Chapter 4 for 
details). .Since the Assess- 
ment's aim is to describe re- 1 
suits for groups* of students 
(males, blacks, students in 
the West, and. so on), not in- 
dividuals, * it is not necessary 
for each student to respond to 
all the items. Each student 
responded to one booklet of 
^items designed to be completed 
in a single class period. 

The 1978-79* assessment was a 
combined assessment of art, 
music and writing. Because of 
the' length of many of the* art 
and writing exercises, only 
two subject ^reas were in- 
cluded in an item booklet. 
- Booklets of exercises included 
either music and writing exer- 
cises, art* and writing exer- 
cises', or art exercises only. 

Following the' selection of art 
exercises, National Assessment 
staff grouped «and sequenced 
them into exercise booklets. 
Since students at different 
ages received somewhat differ- 
ent sets of exercises, book- 
lets -were constructed sepa- 



rately for each age level. 
Thus, exercises for 9-year- 
olds were not sequenced/ in the 
same order as those for 13- 
yearr-olds, and so forth. 

In 1978-79, there were three 
exercise booklets that con- 
tained art exercises for 9- 
year-olds, four such booklets 
for 13-year-olds and five such 
booklets for" 17-year-olds. One 
booklet for 13-year-olds in 
the 197^-80 assessment in- 
cluded art items. 

The following constraints were 
observed in preparing exercise 
booklets : 

1. Each booklet contained 
exercises of varying dif- 

• ficulty so that students 
would not become bored by 
many easy exercises or 
discouraged by many diffi- 
cult exercises. 

2. Exercises could not cue 
othfer exercises. "In other 
words, the answer to one 
exercise could not be con- 
tained in another exercise 
in the same booklet. 

3. Each bookLet was timed so 
that it would take no more 
than '45 minutes of a stu- 
dent's time — the length 
of a typical class period. 
Booklets contained approx- 



imately 30-35 minutes of 
exercise time and an addi- 
tional 10-15 minutes of 
introductory material , 
instructions w and back- 
ground questions, 

4. Booklets were designed to 
be, 'insofar as possible f 
parallel with respect to 
the number of: different 
objectives measured and 
difficulty levels. Items 
measuring f a particular 
objective were scattered 
throughout the booklets so 
that many different stu- 
dents would respond to 
questions related to a 
particular objective. 

5. At ages 13 and 17, book- 
lets containing reassessed 
items used to measure 
change between 1974-75 and 

1978- 79 were identical in 
'content and, item sequence 

in; both assessments. The 
six exercises fo,r 13- 
year-olds assessed in 

1979- 80 (the year follow- 
ing the main art assess- 
ment) were in the same 
booklet. None of these 
items was used to measure 
change in achievement from 
1974-75. 

National Assessment makes 
every effort to minimize dif- 
ficulties connected with the 
testing situation so that re- 
sults will be, as nearly as 
possible, an accurate reflec- 
tion of what students know fand 
can do. For example, students 
marked their answers directly 
in the assessment " booklets, 
not on separate answer sheets. 
It was felt that this proce- 
dure would reduce possibili- 



ties for errors in marking 
answer sheets, especially for 
younger ' students. To minimize 
guessing, students were en- 
couraged a to select the "I 
don 1 1 know" response" option 
included with some multiple- 
choice items or to write "I 
don't know" on the answer line 
for open-ended questions if 
they felt they did not know 
the answer to a question. Many 
multiple-choice art exercises 
asked for a student's opinion 
about a work of art. Such 
items did not include the "I 

don't know" option. 

✓» 

Paced audio tapes were pre- 
pared for each exercise book- 
let. Instructions, most of 
the written portions of an 
exercise stimulus and response 
options we re read aloud to 
minimize the effect of any 
reading difficulties and to 
ensure that all students moved 
through the booklets at the 
same v speed. During the field 
testing of the exercises, ad- 
ministrators had determined 
the time needed for most stu- 
dents to respond to an item, 
and this amount of time was 
allowed on the audio tape. In 
addition , the use of tapes 
helped to ensure uniform as- 
sessment conditions across the 
country. 



Differences in Item Booklets 
in the 1974-75 and 1978-79 
Assessments 

National Assessment makes 
every effort to make assess- 
ment conditions for items 
measuring change * identical 
from assessment to assessment 
so that any changes observed 
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will be attributable to' 
changes in achievement rather 
than a response to an altered 
testing condition. 

As noted previously, booklets 
of items , used to measure 
changes in achievement for 13- 
-and 17-year-olds - included 
identical itemsv in the same 
sequence in both assessments. 
Items used for the first time 
in 1978-79 were included in 
other booklets. Tapescripts 
for the audio tapes accompany- 
ing the booklets used to meas- 
ure change between assessments 
were also identical in each 
assessment. At age 9, book- 



lets including items used to 
measure " change also contained 
newly developed items; there- 
fore, booklets were not the 
same for the two assessments. 
However, individual items used 
to measure ^change and tape- 
scripts for those items were 
identical in both assessments. 
It was felt that the differ- 
ence in item context probably 
would not have any appreciable 
effect on results. Items for 
13-year-olds included in the 
1979-80 assessment were newly 
developed for the second as- 
sessment and thus were not 
used vto measure changes in 
performance. 
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CHAPTER 4 
" SAMPLING 



This chapter give? aa overview 
of the" procedures used in de- 
signing and selecting the sam- 
ples for the 1974-75 and 
1978-79 a^t assessments. Sam- 
ple design and selection for 
both assessments were con- 
ducted by the staff of the 
Research Triangle Institute, 
Raleigh, North Carolina, and 
monitored by National Assess- 
ment staff. , 

The target populations for the 
1978-79 assessment consisted 
of 9-, 13- and 17-year-olds x 
enrolled in either public or 
private schools at the time of 
the assessment who were not 
functionally handicapped to 
the extent that they could not 
participate in an assessment? 
Other specific groups excluded 
we res non-English-speaking 
persons, those identified as 
nonreaders, persons physically 
and mentally unable to re- 
spond, and persons in institu- 
tions or attending schools 



* definitions of 1978-79 as- 
sessment age groups are: 9- 
year-olds — born during the 
calendar year 1969; 13-year- 
olds — born during the calen- 
dar year 1965; and 17-year- 
olds — born October 1, 1961, 
through September 30, 1962. 



established for the physically . 
or mentally handicapped,. 

National Assessment d id not 
follow up specific individuals 
from one assessment to the 
next. 5 In other words, the stu- 
dents who participated in the 
1974-75 assessment we re not 
the same ones who participated 
in 1978-79. However, in each 
assessment year, participants 
were carefully selected to 
. represent each age level. For 
exampl e , a lthough d i f f erent 
sets of probability samples of 
9-year-olds were used for the 
two assessments, each set con- 
tained nationally representa- 
tive samples. of students who 
were 9 years old during that 
assessment year. Thus, if we 
say that 9-year-qlds' achieve- 
ment declined between 1975 and w 
1979, we mean that students 
who were 9 years old in 1975" 
correctly answered the same 
questions more often than 
those who were 9 years old in 
1979., 

The definitions of the in- 
school target populations were 
identical in each assessment. 
The National Assessment sam- 
ples were designed to provide 
approximately 2,500 respon- 
dents per exercise. These num- 
bers allow reporting of data 
for the nation and for the 



subgroups defined in Appendix 
A ii 

There were minor differences 
in the sample designs for the 
1974-75 and 1978-79 art as- 
sessments. A major difference 
in the samples obtained was 
that 17-year-olds who were 
dropouts or early graduates 
were assessed in 1974-75 but 
not in 1978-79, because^ of 
budget limitations. To fensure 
comparability, all comparisons 
of 17-year-olds 1 . 1974-75 and 
1978-79 results were made 
using data only for students 
attending school. All other 
differences are technical im- 
provements or changes that 
increased sampling efficiency 
and lowered administrative 
costs, but did "hot affoct com- 
parability of the samples. 



Overview of the Rational 
Assessment Sample Design 

For all of its assessments, 
National Assessment uses a 
deeply stratified, three-stage 
national „ probability sample 
design with oversampling o£ 
low-income and rural areas. In 
the- first stage, the United 
States is divided Into geo- 
graphical units of counties or 
groups of contiguous cpunties 
meeting a minimum population 
sizq requirement. These 
units, called primary sampling 
units (PSUs), are stratified 
by region and size of commu- 
nity. From the list of PSUs, a 
sample of PSUs is drawn With w 
out replacement with probabil- 
ity proportional to population 
size measures, representing 
all regions and sizes-of com- 
munities. Oversampling of 



low-income and extreme-rural 
areas is first performed at 
this stage by adjusting the. 
estimated population &£ze 
measures cf those areas to 
^increase ^ sampling rates. 
Within PSUs, Census Employment 
Survey Data are used to delin- 
eate and oversample low-income 
areas. Counties with high pro- 
portions of rural families are 
also oversampled. 

In the second stage, all pub- 
lic and private schools within 
-each PSU selected in the first 
stage are listed. Schools 
within e^ach PSU are selected 
without replacement with prob- 
abilities proportional to the 
number of age-eligibles in the 
school. 

The third stage of sampling 
occurs during the data collec- 
tion period, A list of all 
age-el ig ible students within 
each selected school- is made. 
A simple random selection of 
eligible students without re- 
placement is • obtained, and 
item booklets are administered 
to the selected students. Spe- 
cially trained personnel se- 
lect the sample and administer 
the booklets. * 



Survey , Weights 

The number of PSUs, schools 
within PSUs and students 
within schools is determined 
by optimum sampling 
principles. That is; a sample 
design is utilized that 
attempts to .achieve the 
maximum precision for a given 
level of resources. Table 3 
displays the number of PSUs 
used and the number of schools 
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Age 9 
Age 13 
Age 17 



TABLE 3. Nurabar of PSUi and Schools Wi'thin PSUs 
Selected in 1974-75 and 1978-79 



1974-75 
Assessment 
No. of No. of 

PSUs Schools 



115 
115 
115 



1,103 
972 
830 



1978-79 
Assessment 
No. of No. of 

PSUs Schools 



75 
75 
75 



648 
650 
534 



in which assessment sessions 
were conducted, by age, for 
the 1974-75 and 1978t79 
assessments. Appendix C gives 
information about the number 
of students assessed. 



Each respondent in the sample 
did not have the same proba- 
bility of selection because 
some subpopulations were over- 
sampled and because adjust- 
ments were made to compensate 



for some schools' refusals to 
participate and for student 
nonr'esponse. The selection 
probability for each individu- 
al was computed, anO its re- 
ciprocal was used to weight 
each response in any statisti- 
cal calculation to compensate 
for unequal rates of sampling 
and to ensure proper represen- 
tation in the population 
structure. Procedures used to 
assign weights are discussed 
in Chapter 7 and Appendix E. 
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CHARTER 5 
DATA 6%t|ESTI0N 



National Assessment subcon- 
tracted data collection to the 
Research Triangle Institute f 
Raleigh, North Carolina. A 
professional data collection 
staff was used rather than 
school personnel to minimize 
the .burden on participating 
schools and to ensure, insofar 
as possible, uniform adminis- 
trative conditions acros? the 
country ( Final Report. ..In- 
School Field Operations.*. , 
1979) . 

Participation in the National 
Assessment is voluntary. NAEP 
makes every effort to encour- 
age the schools selected in 
the sample to participate in 
the assessment, and National 
Assessment and Research Trian- 
gle Institute staffs have ob- 
tained high rates of school 
cooperation, as shown in Table 
4 (Final Report. . .Field Opera- 



TABLE 4, School Cooperation 
Rates, 1978-79 Assessment 



Age 

9 
13 
17 

Overall 



Percent of Eligible 
Schools Participating 
in <1978-79 Assessment 

90.4 
90.9 
92.9 
91.3 



tions. . . , P9 
27). Stud 
rates were a 
feet of stude 
discussed in 
5 shows the 
students tha 
particular ex 
each age lev 
assessment. 



79, p. 39, Table 
ent cooperation 
lso high. The ef- 
nt nonresponse is 
Appendix C. Table 
actual number of 
t responded to a 
erclse booklet at 
el in the 1978-79 



Each age group was assessed at 
approximately the same time of 
the school year in each as- 
sessment. As noted previously, 
x 3-year-olds were assessed in 
October-Decenber , 9-year-olds 
in January-February and 17- 
year-olds in March-May. 

In 1978-79, booklets were ad- 
ministered to groups of 10-25 
students, with each group re- 
sponding to only one of the 
booklets for their age level. 
The groups varied in size de- 
pending on the number of eli- 
gible students and on an esti- 
mate of the rate of nonre- 
sponse for a particular 
school. In 1974-75, the plan- 
ned session sizes w^re fixed 
at 12 students, with up to 4 
alternates used if less than 
12 students appeared for the 
session. . 

t> 

In each assessment. National 
Assessment takes steps to 
guarantee the anonymity of 
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TABLE 5* Number of Students Responding to Each 
Item Booklet in 1978-79 Assessment, by Age 



Age 9 

Number 
Booklet Responding 



Age 13* 
Number 
Booklet Responding 



Age 17 

Number 
Booklet Responding 



1+ 


2, 532 


1+ 


2,755 


1+ - 


2,730 




£ 1 D D J 




o q m 
z i o Ul 




1 1 AC 


3+ 


2, 475 


3+ 


2,775 


3+ 


2,761 


4+ 


2,494 


4+ 


2,791 


4+ 


2,772 


5+ 


2,479 


5+ 


2,785 


5+ 


2,684 


6+ 


2,522 


6+ 


2,748 


* 6+ 


2,739 


7+ 


2,531 


7+ 


2,736 


7+ 


2,642 


8+ 


2,5,24 


8+ 


2,779 


8+ 


2,656 


9 ' 


2,486 


9+ 


' 2,754 


9++ 


2,787 


10 


2,483 


10 


2,758 


10+ 


2,697 


11 


2,526 


11 


2,751 


11 


2,628 






• 12 


2,720 


12 


2,628 






13 


2,757 


13 


2,698 










14 


2,654 


Total 


27,605 




35,910 




37,822 




assessment , several 

oklet 12, which was taken by 2,748 



age 



13 



art items 



*During the 1979 
were included irf 
respo ndents. 

+Thefre were no art exercises- included in booklets 1 through 
8 at age 9, booklets 1 through 9 at agg 13 and booklets 1 
through 8 and^lO at age 17. 

++OnIy exercise 6 of booklet 9 at age 17 was an art 
exercise; it is a combined art/writing, exercise . 



each- .respondent. Students 1 
names " were listed with their 
booklet identif icatibn number 
so that scoring and processing 
personnel could go back to the 
school lists for data verifi- 
cation — for instance , on 
background information — if 
necessary. These lists did not 
leave the schools and were 
destroyed six months following 
the assessment in a school. 

To provide information on re- 
spondents* backgrounds, school 



officials were asked to re- 
spond to a "principal 1 s ques- 
tionnaire," which included 
questions about the size and 
type of community served by 
the schools. In addition, in 
1978-79 officials in schools 
were asked to respond to a 
"supplementary principal f s 
quest ionnai re , 9 which asked 
about art, music and 
writing/language arts programs 
in the school. Students also 
provided information on their 
backgrounds through questions 
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included in the item booklets. 
Samples of forms used .to col- 
lect background information 
from students and school offi- 
cials in the 1978-79 assess- 
ment appear in Appendix B. 

The assessment administrator 
cod^i each ■ student's . birth 
date, /sex, c,rade and 
racial/ethnic classification 
on his or her booklet. Iden- 
tification numbers were pre- 
coded by machine on. the front 
of each booklet. Administra- 
tors made a* visual racial/eth- 
nic identification at the time 
each student turned in his or 
her booklet. During the 1978- 
79 assessmet/ - six different 
racial classifications were 
used: white, black, Spanish 
heritage, American Indian or 
Alaskan native, Pacific Is- 
. lander or Asian, *nd unclassi- 
fiod. If an administrator was 
unsure of a student's 
racial/ethnic^group, he or she 
referred to the, student's name 
or questioned school personnel 
to make the identification.. 
In a few cases* listening to 
the student's speech was an 
aid. The assessment adminis- 
trator did not ask students to 
give a racial identification 
for themselves; however, in 
1978-79 17-year-old stents 
were .asked* to provide this 
.information in one of the 
background questions included 
in the exercise booklet. 

Sample sizes of the two clas- 



sifications American Indian 'or 
Araskan native- and Pacific 
Islander or t Asian, are too 
small to permit reporting for 
these groups. Results for- the 
group classified as Spanish' 
heritage cannot be reported 
for separate exercises but can 
be reported for aggregate re- 
sults across a number of exer- 
cises. 

Following data collection,, 
assessment administrators sent 
completed booklets to the 
scoring contractor, Westing- 
house DataScore Systems, Iowa 
City, Iowa . Booklets were 
quality-checked to verify that 
correct administrative proce- 
dures were followed by the 
field staff. Coded identifi- 
cation information was also 
checked for .accuracy; incon- 
sistencies that could not be 
reconciled were sent back to 
the assessment administrator 
to be checked against the list 
of student names and identifi- 
cation numbers retained by the 
school for six months follow- 
ing the assessment. 

In 1974-75, 17-year-olds who 
were not . currently attending 
school were included in the 
assessment. These out-of- 
school 17-year-olds could each 
answer up to, four booklets qf 
assessment materials; they 
were paid $5.00 for each book- 
let they completed. Unpaced 
audio tapes were used for this 
group. 
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CHAPTER- 6 
SCORING 



Scoring and computer recording 
of" data were contracted to 
Westinghouse DataScore Sys- 
tems', Iowa City, Iowa. While 
the majority of exercises in 
the 1978-79 art assessment 
were multiple-choice, several 
exercises were open-ended* 
Responses to < multiple-choice 
exercises were read directly 
from the booklets by optical 
scanning machines . .The scoring 
contractor employed a special 
staff to hand score the open- 
ended exercises, Scorers were 
responsible for categorizing 
responses, using the scoring 
guides fo?: open-ended exer- 
cises that defined categories 
of acceptable and unacceptable 
responses. They then coded 
this information into ovals 
that could be read by optical 
scanning machines. 

Because of the complexity and 
expense of art scoring, the 
six open-ended art exercises 
scheduled to be scored follow- 
ing the second (1978-79) art 
assessment were .not scored 
until fall 1980. All of these 
exercises had been used in the 
1974-75 assessment as well as 
during the 1978-79 assessment. 
Responses from the first as- 
sessment had been held un- 
Scored so that responses from 
both assessments could be 
scored at the sam§ time, by the 
same scorers* Similarly, some 



open-ended un released items 
from the 1978-79 assessment 
remained unscored so that re- 
sponses can be scored with 
responses from a subsequent 
assessment. 

The art exercises were scored 
using scoring guides that had 
been created by art consult- 
ants, using as a* reference 
field trial responses from the 
field trials completed prior 
to the 1.974-75 assessment. 
These guides we re edited and 
revised by National Assessment 
staff. Ih September 1980, 
staff from NAEP and. Westing- 
house DataScore Systems worked 
ta refine these scoring guides 
through trial juse with a sam- 
ple of responses to each exer- 
cise from the 1978-79 assess- 
ment. The Westinghouse staff 
included an artist retained as 
a consultant for this scoring 
project and several experi- 
enced scorers. Following re- 
finement of the six scoring 
guides, this team of scorers 
assigned scores to a set of 
training papers. On more than 
one occasion, art consultants 
who had worked with the ear- 
lier versions of the guides 
were contacted for clarifica- 
tion. 

Of the six open-ended exer-< 
cises scored , two exercises 
required short, written an- 
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swers and were administered to 
all three ages. Four items 
asked for production of some 
kind of drawing. Two of these 
"drawing 1 * items were adminis^ 
tered to all three ages; two 
were administered only to 13- 
and 17-year-olds. 

Items were scored by eight 
scorers organized into two 
teams of four . In addition/ 
the artist consultant worked 
with both teams ^as the art 
authority on scoring questions 
and did some scoring. Each 
scoring team was responsible 
for the scoring of one 
short-answer and two art 
production exercises. Thus, 
each scorer had to master only 
three scoring guides. Since 
the scoring guides varied from 
about 15 pages for the 
short-answer exercises to over 
100 pages for the art 
production exercises, it was 
felt that moWL tjian three 
, guides would be^excessively 
difficult to learn. 

During training of the scor- 
ers, - conducted in October 
1980, scoring guides were dis- 
cussed, and scorers then began 
working with small sets of 
training papers. Their scores 
and the master scores assigned 
during scoring guide refine- 
ment were compared . and dis- 
crepancies were discussed. In 
a number of, cases, inconsis- 
tencies in the master scores 
were detected and corrected. 
These discussions led to some 
further scoring guide revi- 
sions and refinements. 

When scorers apd trainers were 
satisfied that the scoring 
guides could be consistently 
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applied, the scoring of the 
actual responses from the 
- 1974-75 and 1978-79 assess- 
ments began. To help ensure 
that the guides were uniformly, 
used across years and across* 
ages, scoring of responses 
from all three ages and from- 
both assessments proceeded 
simultaneously. 

To be sure that the art hand 
scoring was reliable across 
the scoring period and across 
scorers , National .Assessment 
asked Westing house DataScore 
Systems to pe rf o rm two 
quality-control studies. In 
one study, a sample of about 
3% of the responses to one of 
the six open-ended questions 
was_ drawn during each week of 
scoring. Each of these re- 
sponses was scored by two 
scorers who were normally as- 
signed to score the item. The 
two scorers independently 
rated ^ each response and their 
scores were then compared. As 
seen in Table* 6, the overall 
pairwise percentage of agree- 
ment was 94.6% — Team 1 aver- 
aged 94.9% and Team 2 averaged 
94.4%. 



In the second quality-control 
study, which began near the 
beginning of the scoring pe- 
riod, scorers scored a sample 
set of responses; they then 
rescored the same set of exer- 
cises at the conclusion of the 
scoring period. * The two 
scores were compared and pair- 
wise percentages of agreement 
were calculated , this time 
within scorer and across time. 
Team 1 averaged 92.5% and Team 
2 averaged 95.2% agreement 
(see Table 7) . 
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TABLE 6. Between-Scorer Quality-Control Summaries 
for Art Open-Ended Scoring Summarized Across Seven Weeks 

Exercise # # of Scores # of Agreements % of Agreement 

Team 1 „ QC , 

.101005 2,520 2,399 95.2 

301008 3,240 3,046 94.0 

303042 4,800 4,574 95.2 



Total 



10,-560 10,019 94.9 



302006- 14,280 13,566 9,5.0 

304014 6,900 6,521 94.5 

501012 ' 1,680 1,495- *9.0 

Total 22,860 21,582 94.4 

Grand total 33,420 31,601 94.6 



TABLE 7. .Wi thin-Scorer Quality-Control 
Summaries for Art Open-Ended Scoring 
(Readings at Beginning and Conclusion of Scoring^ 

Ixtroift * # of Scores # of Agreements % of Agreement 

Tearo 1 on p 

101005 720 654 90. 8, . 

301008 1,080 987 91.4 

303042.. 



Total 



1,680 1,577 93.9 

3,480 3,218 92.5 



T 302006 - 3,060 2,939 96-0 



304014 2,070 
501012 360 



1,961 94.7 
327 ' 90.8 



Total 5,490 5,227 , 95.2 

8,445 94.1 



Grand total 8,970 



The results of the ongoing 
quality-control study were 
closely monitored by Westing- 
house. When necessary, correc- 



tive actions were taken, in- 
cluding retraining and some 
rescoring of work units al- 
ready completed. 
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CHAPTER 7 
DATA ANALYSIS 



Measures of Achievement 

The basic measure of achieve- 
ment reported by National As- 
sessment is the percentage of 
students responding acceptably 
to a given item. This percent- 
age is an estimate of the per- 
centage of 9-, 13- Or 17- 
year-olds who would respond 
acceptably to a given item if 
every 9-, 13- or 17-year-old 
in the country were assessed. 

Percentages of acceptable re- 
sponses are used because each 
- item is designed as a separate 
measure of some aspect of an 
objective or subobjective. The. 
purpose of National Assessment 
is to discover if more or 
fewer people are able to an- 
swer these items acceptably — 
and thus meet the objectives 
— over time. 

In addition to providing re- 
sults on individual items, 
National Assessment reports 
the average performance across 
groups of similar items, for 
example, the learning area as 
a whole, for particular ob3ec- 
tives or subobjectives, and so 
forth. The results- are the 
mean, or arithmetic average, 
of the estimates, of perform- 
ance on the gtoup of items, 
which is caj^eti t^e mean -per- 



centage acceptable. 1 The exer- 
cises included in the calcula- 
tion Of a mean percentage are 
Usually located in several 
exercise booklets, and the 
same booklets are not adminis- 
tered to all students. 

To present a general picture 
of changes in achievement, 



iTwenty-two empirical distri- 
butions of change measures 
from the 1969-70 and 1972-73 
science assessments were used 
to generate Monte Carlo simu- 
lations of sampling distribu- 
tions for several measures of 
central, location. In addition 
to the mean and median, other 
measures of central location 
that were considered in the 
simulation studies included 
the average of the extremes, 
two forms of biweighted esti- 
mates and three forms of 
weight-matching estimation 
described by John W. Tukey in 
the research paper "Some Con- 
siderations on Locators 'Apt 
for Some Squeezed-Tail (and 
" Stretched-Tail) Parents" 
(1975). In almost every case^ 
the sampling stability of the 
mean change was as good as or 
* better than that of the other 
measures studied. 



National Assessment describes 
the gains* and * losses on a 
group of exercises in terms of 
the differences in the average 
percentages of acceptable re- 
sponses*. 

Unless the exercises summar- 
ized in the mean percentages 
of acceptable responses are 
identical, the means^ of one 
age group should not^be com- 
pared with the means of an- 
other? since their values re- 
flect both the choipe of exer- 
cises and the performance of 
the students « When only a few 
exercises are summarized by a 
mean, one should be especially 
cautious in interpreting, re- 
sults, since a^small set of 
exercises might not adequately 
cover the wide range of poten- 
tial behaviors included under 
a given objective or subobjec- 
tive. The mean should be in- 
terpreted . literally as the 
arithmetic average of the per- 
centage of acceptable respons- 
es obtained from National As- 
sessment samples on a specific 
set of exercises. 

In addition to providing na- 
tional results, NAEP reports 
"the achievement of various 
subpopulations of interest. 
Groups are defined by region 
of the country, sex, race/eth- 
nicity, 'size of community, 
type of community, grade, and 
level of parents 1 education. 
Results from some additional 
variables are also analyzed. 
The definitions of reporting 
groups are found in Appendix 
A. Forms used to obtain back- 
ground .information are pre- 
sented in Appendix B. 

In considering National As- 



sessments achievement meas-^ 
yures, it is the differences in 
performance between assess- 
ments) among groups and among 
ages that are the most useful* 
By maintaining the same item 
or set of items in making 
these compa'r isonsi, we have a 
reasonable indicator of 

whether more or fewer people 
know or can do something 
judged important. 



Procedures for estimating per- 
centages of acceptable re- 
sponses to exercises are de- 
pendent on the sample design. 
Each response by an individual 
is weighted and multiplied by 
an adjustment factor for non- 
response* 2 An estimate of the 
percentage of the group that 
would have responded to an 
exercise acceptably if the 
entire age group were assessed 
is defined as the weighted 
number of acceptable responses 
divided by the weighted number 
of all the responses. A simi- 
lar ratio of weights is used 
to estimate percentages . of 
acceptable responses for re- 
porting groups or subpopula- 
tions of interest. 



^Appendix C discusses nonre- 
sponse in assessment samples. 

v ^A weighting-class adjustment 
procedure was used to smooth 
estimated population * propor- 
tions across the assessments 
conducted between 1970-71 and 
1979-80. Appendix E discusses 
this procedure. 
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Estimating Variability in 
Achievement Measures 

National Assessment,, uses a „ 
National probability sample at 
each age level to estimate the. 
proportion of people who would 
-complete an exercise in'a .cer- 
tain way."* The particular sam- 
ple, selected is one of a large 
number of all possible samples 
of the same size that could 
have been selected with the 
same sample design. Since ain^ 
achievement measure computed 
from each ot the possible sam- 
ples would, differ -from , one 
sample to another, the stan- 
uard error of this statistic is 
used as a measure of the sam- 
pling • variability among 
achievement measures from all 
possible samples. h standard 
error, based on one particular 
sample, serves to estimate 
that sampling variability. 

In the interest of sampling- 
and cost efficiencies,, Na- 
tional Assessment uses a com- 
plex, stratified, multistage 
probability sample design. 
Typically, complex designs do 
not provide for unbiased or 
simple computation of sampling 
errors. A reasonably good ap- 
proximation of standard error 
estimates of acceptable re- 
sponse percentages is obtained 
by applying the jackknife pro^ 
cedure (Miller, 1964, pp. 
1594-1705; Wilier, 1968, pp. 
567-582; Mosteller and Tukey, 
1968) to first-stage sampling 
units within strata. Standard 
errors for achievement meas- 
ures such as group differ- 
ences, mean percentages or 
mean group differences for a 
particular assessment year are 
estimated directly, taking 



advantage of features of the 
jackknife procedures that are 
generic to all of these sta- 
tistics. 4 Since samples for - 
different \ assessments are in- 
dependent, „the standard errors 
of the differences in achieve- 
ment measures between assess- 
ments can ±>e estimated simply 
by -the square root of the sum 
cf " squared standard errors 
from each of the assessments. 

The "standard error provides an 
estimate of sampling reliabil- 
ity for the achievement meas- 
ures used by National Assess- 
ment. It is comprised of sam- 
pling error and other random 
error associated with the as- 
sessment of a specific item or 
set of items. Random error 
includes all possible nonsys- 
tematic error associated with 
administering specific exer- 
cises to specific students in 
specific situations. For 
open-ended items, random dif- 
ferences among scorers are 
also included in the standard 
errors. 

National Assessment has ad- 
, hered to a standard convention 
whereby differences between 
statistics are designated as 
statistically significant at 
the .05 level of significance. 
That is, differences in per- 
formance between assessed 
years or between a reporting 
group and the nation are high- 



4 See Appendix D for a more 
detailed description of Na- 
tional Assessment's' computa- 
tion of standard errors. 
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lighted with asterisks only if 
they are at" least twice as 
large as their standard error. 
Differences this large would 
occur by chance in fewer than 
5%T of all possible replica- 
tions of the sampling and data 
collection procedures for any 
particular reporting group or 
national estimates. 



Controlling Nonrandom 
* Errors 

Systematic errors can be 
introduced at any stage of an 
assessment — exercise devel- 
opment, preparation of exer- 
cise booklets , design or ad- 
ministrative procedures, field 
administration, scoring or 
analysis. These ncnsampling, 
nonrandom errors rarely can be 
•quantified, nor can the magni- 
tude of the bias they intro- 
duce into the .estimates be 
evaluated directly. 

» Systematic errors can be con- 
trolled in large part by em- 
ploying uniform administrative 
and scoring procedures and by 
requiring rigorous- quality 
control in all phases of an 
assessment. If the systematic 



errors are the same from age 
to age or group to group, then 
the difference in percentages 
or mean percentages are meas- 
ures with reduced' bias because 
subtraction tends to cancel 
the effect of the systematic 
errors. 

Similarly, the effect of 
systematic errors in different 
assessment years can be 



controlled by 
r epl ica t ing in 
assessment the 
carried out in 
Differences in 
across assessment 



carefully 
the second 
procedures 
the . first, 
achievement 
years will 



also be measures with reduced 
bias since subtraction will 
ag^in tend to cancel 
sys 6matic errors. 

Although it is not possible 
for every condition or proce- 
dure to remain exactly the 
same between assessments" con- 
ducted several years apart, 
National Assessment makes 
every effort to keep condi- 
tions as nearly the same as 
possible. Changes in proce- 
dures described in this report 
were judged to have a rela- 
tively minor impact. 
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APPENDIX A 

DEFINITIONS OP NATIONAL ASSESSMENT REPORTING GROUPS 



In addition, to reporting 
results for all "9-, 13- and 
17-year-old 1 students in the 
United States, , National 
Assessment reports results for 
a number of population 
subgroups. These subgroups are 
defined for both the 1974-75 
and the 1978-79 assessments. 



Group Definitions 

Definitions of the subgroups 
foliow: 

Region 

The country has been divided 
into four regions: Northeast , 
Southeast, Central and West. 
States included in each region 
are shown on the following 
map. 




Sex 

Results are' reported for males 
and females. 

Race/Ethnicity 

Results are presented for 
blacks, whites and Hispanos. 
(Because of* "a small sample 
size, only average results for 
Hispanos can be reported by 
National Assessment.) 



Level of 

Parental Education 

National Assessment defines 
three categories of parental- 
education levels, based on 
students' reports. These cate- 
gories are: (1) those whose 
parents did not graduate from 
hich school, (2) those who 
have at. least one parent who 
araduated from high school and 
(3) those who have at least 
one parent who has had some 
post-high-school education. 



Type of Community 

Communities in this category 
are defined by an occupational 
profile of the area served by 
a school as well as by the 
size of the community. in which 
the school is located. This 
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is the only reporting group 
♦that excludes a large number 
of respondents ♦ About two- 
thirds do f not fall A into the 
classifications listed below* 
Results fo'r the remaining 
two-thirds, are Hot reported 
since their performance is" 
similar to that of the nations 

Advantaged-urban (high-metro) 
communities . Students in this 
group attend schools in or 
around cities having a popula- 
tion greater than 200,000 
where a high proportion of the 
residents are in* prof essional 
or managerial positions, 

Disadyantaqed-urban (low- 
metro) communities . Students 
Tn this group attend schoo 1 s 
in or around cities having a 
population greater than 
200,000 where a 'relatively 
high proportion of the resi- 
dents are on welfare or^are 
not regularly employed . 

Rural communities , • students in 
this group attend schools in 
areas with a population under 
10,000 where many of the resi- 
dents are farmers or farm 
workers. 



Size of Community 

Big cities . Students in this 
group attend schools within 
the city limits of cities hav- 
ing a 1970 census population 
over 200,000. 

Fringes around big cities . 
Students in this group attend 
schools within metropolitan 
areas (1970 U.S. Bureau of the 
Census urbanized areas) served 
by cities having a population 



greater than 200,000 but out- 
side the city limits. 

Medium cities .^ 'Students in 
this group attend schools in 
cities having a population 
between 25,000 and 200,000, 
not classified in the 
fringes-around-big-cities cat- 
egory. 

Small places . Students in this 
group attend schools in commu- 
nities having a population 
less than 25,000, not classi- 
fied in the f r inges-around- 
big-cities category. 



Grade in School 

Results are categorized for 
9-year-olds in grades 3 and 4; 
13-year-olds in grades 7 and 
8; and 17-y.ear-olds in grades 
10, 11 and'l2. 



Modal Grade by Region 

Results are categorized for 
■9-, 13- and 17-year-old re- 
spondents in grades 4, 8 and 
11, respectively, who live * 
the Northeastern, Southeas. 
ern, Central or Western region 
of the country. 



Modal Grade by 
Community Size 

Results are categorized for 
9-, 13- and 17-year-old re- 
spondents in grades 4, 8 and 
11, respectively, who live in 
big cities, fringes around big 
cities, medium cities and 
small places. 
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Modal Grade by Sex 

Results are categorized for 
9-, 13- and 17-year-old males 
and females in grades 4, 8 and 
ll 9 respectively. 



by those who reported collect- 
ing none, one type and t;w6 
types of art. Age 17 Jesuits 
are categorized by tHbse who * 
reported collecting none , one 
type, two types and three 
types of art. 



Visit Art Museums 

Results are categorized for 
9-, :13- and 17-year-old re- 
spondents who rep ~ted never 
having visited an - museum, 
having visited an art museum 
once and having visited an art 
museum five or more times. 



Art Taught 

Results are categorized for 
9-, 13- and 17-year-olds whose 
principals reported that their 
schools offered at least one 
art class and for those whose 
principals reported that their 
schools offered no art 
classes • 

Do You Collect Art 

Age 13 results are categorized 



What Kinds of Artwork 
Do You Do 

Age 9 results are categorized 
by those who reported doing 
none, 1 or 2, and 3 or 4 types 
of art outside of school. Age 
13 and 17 results are catego- 
rized by those who reported 
doing none, 1 or 2, 3 or 4', 
and from 5 to 10 types of art 
outside of school. 



Art Classes Taken 

Age 13 results are categorized 
by those who reported taking 
none, one or two art classes. 
Age 17 results are categorizec 
by those who reported takim 
none, one, two or three, o 
four to six art classes. 
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APPENDIX B 



FORMS USED TO OBTAIN BACKGROUND INFORMATION * 

* * • * 

This appendix contains the 1978-79 at^t assessment. Fol- 
forms used by National Assess- lowing is a listing and a 
went to collect background brief description of the forms 
information from school offi- included, 
cials -and respondents in the 

> . / . 

p # 33 . School Principalis 'Questionnaire — filled out by 
school principals or other school officials for 
schools at each of the age levels discussed* 

p. 35 Supplementary Principal 's Questionnaire (Aae ^9) 
— given .to principals of schools in which the 
age 9 assessment took place. The questionnaire 
provides information about the art and music 
programs of the schools. . . 

p. 38 Supplementary Principal's Questionnaire (Age 13) ~ 
— given to principals of schools in which the 
age 13 assessments took place. The questionnaire v 
provides information about the art, music and 
writing or language arts programs of the schools. 

p. 40 Supplementary Principal's Questionnaire (Age 17) 
—given to principals of schools in which the 
age 17 assessments took plaqe. The questionnaire 
provides information about the art, music ? and 
writing or language arts programs of the schools. 

p # 42 Standard Background Information Form for 9-Year- 
Olds — provides, information about reading mate- 
rial in the home and level of parents 1 education. 



*This principal's questionnaire 
was not used with the age 
13 art, exercises given in 
1979-80. 
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p. 43 Standard Background Information Form for 13- 
Year-Olds r- provides information about read'ng 
material in the home, level of parents' educa' on 
and plafce lived in at lage 9. 

p. 44 Standard Background Information Form r 17- 
Year-Olds — provides information on homework, TV 
watching, racial identification, possessions in 
the home and classroom ^activities, in addition to 
questions also asked of 9- and 13-year-olds. 

p. 47 Background Information: Art* Museum or Gallery 
Visits --9-/ 13- and 17 -Year-Olds , This exer- 
cise asked for responses about the number of art 
museums or galleries visited. It was included in 
Jill 9-, 13- and 17-year-olds* art booklets given 
in 1978-79 except booklet 9 at age 17. 2 It was 
not included in the age 13 art materials assessed 
in 1979-80. . 

p. 48 Background Information: Kinds of Artwork Done 
Outside of School — 9-, 13- and 17-Year-01ds . 
These two excercises, one for age 9. and one for 
c^ges T3 and 17, asked respondents * if they had 
done a number of different types of artwork out- 
side of school. The appropriate versions appear 
in r all 1978-79 and 1979-80 booklets except book- 
let r 9 at age 17. 2 

p # 50 Background .Information: Art Classes Taken — .13- 
and 17-rYeaf-Olds ., These two. exercises, one for 
each age, asked about art classes taken or^ pres- 
ently enrolled in. The appropriate version was 
* * included in all 1978-79 and 197&-80 booklets for 
13- and*17-year olds except booklet 9 at' age 17. 2 

p. 52 Background Information: Types of Art Collected — 
; 13- and 17-Year-Oldg . There were two versions of 
this exercise, one E^fi^l3-year-oldS and another 
for 17-year-olds. Tire impropriate version of the 
exercise appeared in. all 1978-79 art booklets for 
13- and 17-year-olds except booklet 9 at age 17. 2 
This exercise was not included with the 1979-80 v 
age 13 art materials. 



2 Booklet 9 for 17-year-oldis 
included only one exercise in 
art. This item was an essay 
item designed to measure, both 
art and writing. 
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School Principal's Questionnaire 



Th» rtporx * •uthorlitd bv I*" <20 U.S.C. 1231 c*1>. WhiH 
you «rt not required to rwpond, your cooptniion t* n««d«d 
to m»kt th« rnults of thu >urv«Y compnhw»iJvt, accunfo. 
*nd umtlv. 



Primary Sampling Unit 



Name of School 



School Number 
Age Group(s) 



13 



'£ 17 



PLEASE 
PRINT 



Address of School 



(Street) 



(City) 

Name of School Principal 



(State) 



(Zip Code) 



Name and title of person completing the form if other than school principal 
Name Title 



1. What is your best estimate of the current enrollment and the average daily 
attendance by grade of your school (1978-79 school year)? (Enter zeros for 
grades not served by your school*) 



Grade 

Enrollment 

Average 

Daily 

Attendance 



K 


1 


2 


3 


A 


5 


6 


7 


8 


9 


10 


11 


12 























































Approximately what percentage of the students attending your school liv^e in each 
of the followir,. areas? 



JK a In a rural area (less than 2,300) 
JS B In a town of 2,500 to 10,000- . ^ 
X C In a town of 10,000 or more 

(Items A-C should add to 100%) 



100% 
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BEST COPY AVAILABLE 



Approximately what percentage of the students attending your school are children 
of 

X A Professional or managerial personnel 

Z B Sales, clerical, technical or skilled workers 

Z C Factory or other blue collar workers 

X D Farm workers 
Z E Persons not regularly employed 

Z F Persons on welfare 



100Z 



(Items A-F should add to 100Z) 



4, Approximately, what percentage of the students attending your .school are 

Z A American Indian or Alaohari Native 

X B Asian or Pacific Islander 



X C Hispanic, regardless of race (Mexican, Puerto Rican, Cuban, Central 
or South American or other Spanish culture or origin) 

JS D Black and not Hispanic 

_Z E White and not Hispanic 

(Items A-E should add to 100Z? 



100Z 



5, Dqes your school qualify for ESEA Title I assistance? 

Ye» - If Yes, approximately what number of students qualify for 

ar.d what number of students are receiving ESEA Title I assistance? 

Approximate number of students qualifying for ESEA 

Title'! assistance 

* Approximate number of students receiving ESEA Title I 

assistance 



No 



THANK YOU FOR YOUR COOPERATION 



34 



41 



Supplementary Principal's Questionnaire (Age 9) 



Instructions: The purpose of this questionnaire is to provide additional information vnich 
will be used »n the analyses of NAEP data. Darken the appropriate ovals 
with asoft lead pencil. IfyouhavequestionsaboutanyofthefolV wingitems, 
please contact the National Assessment district supervisor. Thank you lor 
your cooperation. 

1. Is art taught in your school? 

CD Yes cd No (Go to Question 10 on page 3) 

2. Are there specified and systematically ordered objectives for art instruction in 
your school? 

cp Yes o No 

•\ 

3. Which of the following items are available as art teaching aids in your school? 



A. 


SUides of works of art 


Yes 


No 






CD 




B. 


Films on art 


Yes 


No 








CD 


C. 


Film strips of works of art 


\ es 


No 






D. 


Color reproductions of works of art 


Yes 


No 

CD 


E. 


Art books 


Yes 


No ' 






c=> 


<=?, 


F. 


Original works of art 


Yes 


No* 






CD 


G. 


Other 


Yes 


No 






cd- 





4. Does your school have a special art room? 



cr> tes 



cd No 
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5. Which of the following art equipment is found in your school? 



A. 


Pottery kiln ~ 


Yes 


No 




- CD 


CD 


B. 


Potters wheel 


, * Yes 


No 






cd 


CD 


C. 


Weaving loom 


Yes 


No 




CD 


CD 


D. 


Easels . 


Yesv 


No 






CD 


CD 


E. 


Cameras 


Yes 


No 






CD 


CD 


F 


Other 


Yes 


No 






CD 


CD 



6. Do the nine-year-old students in your school receive art instruction? 
cd Yes cd No (Go to Question 10 on page 3) 



If the nine-year-old students receive art instruction* who does the teaching? 
(Mark each appropriate oval.) 



A. 


The classroom teacher 


J 

Yes 

CD 


No 

CD ' 


B. 

L 


A member of a teaching team^ having 
a background in art but not certified - 
in art 


Yes 

CD 


No 

CD 


d. 


A certified art teacher 


Yes 


No 

CD 


D. 


Other 

s 


Yes 

CD 


No 

CD 



8. On the average, how much time each week is given to art instruction of nine- 
year-old students? 
cd Less than 30 minutes 

cd 30 to 59 rninutes > 
CD 60 to 89 minutes 
cd 90 to 119 minutes 

cd 120 \i inutes and over . 



KLC 
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9. For nine-year-old students, are field trips taken to art museums or art 
galleries? 

CD Yes c=> No * 



r 



,t,I KJC , 



i * 
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Supplementary Principal's Questionnaire (Age 13) 



Instructions: The purpose of this questionnaire is to provide additional information which will 
be usedun the analyses of NAEP data. Darken the appropriate ovals with & soft 
lead pencil. If you have questions about any of the following items, please contact 
the National Assessment District Supervisor. Thank you for your cooperation. 



1. Is art taught in your schpol? 

'O Yes * cd No (Go to Question 12 on page 3) 

2. Are there specified and systematically ordered objectives for art instruction 
in your school? 

cd Yes CD No 

3. Which of the following items are available as art teachingaids in your school? 



4. 



A. 


Slides of works>of art 


Yes 

CD 


No 

CD 


B. 


v Films on art 


Yes 


-No 

CD 


C. 


Filnvstrips of works of art * 


Yes 

CD 


No 

CD " 


D. 


Color reproductions of works of art 


Yes 


No 

CD 


B. 


1 Art books 


Yes 

CD 


1*0 
CD 


F. 


Original works of art 


Yes 


No 

CD 


G. 


Other 


Yes 

CD 


^No 

CD 


Are students required to take art in seventh grade? 






CD 


Yes <=3 No (Go to Question 6) 




/ 


CD 


No seventh grade (Go to Question 8) 







5. If there is a requirement, for what length of time do the students ,take_&r£? 
o 1/4 school year or less ' ~ . 
cd l]f2 school year 

cr» 3/4 school year 
cd Full school year . 

6. Is an elective art course available for seventh gratfe students? 
cd Yes cd No (Go to Question 8) 
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7. If you said "Yes" to Question 6. approximately whatpercentage of seventh 


i 


ife -_" -- 


graders elect to take art cour?es # each year? 
CD 0to24% 

i CD 25 to 49% ^ , 
CD 50 to 74% 
CDf 75 to 89% 
CD 90 to 100% 

8. Are the students required to take art in eighth grade? 




■--iT~- -" V 


O Yes o NoJGo to Question 10 ort page 3) 
CD No eighth grade {Go to Question 12 on page.3) 






o * 9. If there is a requirement? for what lengtfc of tim^do the students take art? 






cd 1/4 school year or less 






cd 1/2 school year 4 


Si 




CD 3/4 school year 
cd Full school year. 






10. Is an elective art course available for eighth grade students? 


_~ * 




cd Yes c£> No (Go to Question 12) v 






11. If you said "Yes" to Question 10, approximately what percentage of eighth 




- 


graders elect to take art courses each year? 






CD 0 to 24% 


* - 


.7*1"-- * 


CD Id to 49% 
CD 50 to. 74% ' 
CD 75t>89% 
CD 90 to 100% 






* 

s 












* t' 
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Supplementary Principal's Questionnaire (Age 17) 



Instructions: Tht purpose of this questionnaire is to provide sdditkmsl information which will 
be used In the analyses of NAEP data. Darken the appropriate orals with a soft 
lead pencil, If you have questions about any of the following items, please contact 
the National Aseemnent district supervisor. Thank you for your cooperation. 



> £ la art taught in ybur school? 

cd Yes cd No (Go to Question 7 on page 3) 

2. Are there specified and systematically ordered objectives for art instruction - 
in your school? 

- t 
cd Yes cd No & 

S. Which of the following items are available as art teaching aids in your school? 



A. 


Slides of works of art 

ft- 


Yet 

CD 


No 

CD 


B. 


Filma on art v * r 


Yei 

CD 


No 

CD 


C. 


Film stripe of works of art 


Ye» 

CD . 


No 

CD 


& 


Color reproductions of works of art 


Ye§ 

CD 


No* 

CD 


E, 


Art books 

> 


Yta 

CD 


No 

CD 


*•« 


' Original works of art 


Yee 

CD 


No 

CD 


6. 


'Othw 


Yei 

CD 


- No 

t CD 



4. Is art instruction available in your school for the following grades? 

No such . ' ] 

— -Grade Yes — — — No — 



A, 


Grade 9 


CD 


CD 


CD 


B. 


Grade 10 


CD 


CD 


» CD 


C. 


Grade 11 


CD 


CD 


CD 


D. 


Grade 12 


» CD 


CD 


CD , 


£. 


Ungraded 


CD 


CD 


CD 
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5. Doe. your echool offer the following types of art instruction for Grtdei 9, 10, 
11 or 12? 



fa 




Yes 


No 


A. 


General art 


r " j 


CD 


B. 


Pine art 


CD 


CD 


C. 


Crafts 


CD 


CD 


D. 


Humanities • 


CD 


CD 


IB. 


Art history 


. CD 


CD o 


F. 


Commercial art 


CD 


CD 


G. 


Other 


CD 


CD 



About what percent of the 17*year-old students are currently enrolled in art 
classes? 

cd 0 to 24% . 
CD 25 to 49% . 
CD 50 to 74% 
CD 75 to 89% 
CD . 90 to 100% 
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BEST COPY AVAILABLE 



Standard Background Information Form for 9~Year~Olds 



1. Does your family get a newspaper regularly? 
O Yes O No o I don't know. 

2. Does your family get any magazines regularly? 
CD Yes CD No CD I don't know. 

3. Are there more than 25 books in your home? 
O Yes o No o I don't know. 

4. Is there an encyclopedia in your home? 

CD Yes CD No CD I don't know. 

5^ How much school did your father complete? 

(FILL IN THE ONE OVAL which best shows how much school your 
father completed.) 

O Did not complete the 8thVade 

CD Completed the 8th grade, but did not go to high school 

O Went to high school, but did not graduate from high school 

O Graduated from high school 

O Some education after graduation from high school 

CD I don't know. 

6. Did your father graduate #bm a college gt university? ' 
CD Yes o No O I don't know. 

7. How much school did your mother complete? 

(FILL IN THE ONE OVAL which best shows how much school your 
mother completed.) 

CD Did not complete the Sth grade 

CD Completed the 8th grade, but did not go to high school 

~ ~CD Went to high school, out dfd not graduatel^ron^ ~™ 



O Graduated from high school 

CD Some education after graduation from high school 

CD I don't know. 

8. Did yoi*r mother graduate from a college or university? 
CD Yes CD No CD .1 don't know. 
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Standard Background Information Form for 13-Year-Olds 



1. Does your family get a newspaper regularly? 
a y« o No rzz I don't know. 

2. Does your family got any magazines regularly? 
CD Yes CD No o- I don't know. 

3. Are there more than 25 books in your home? 
i — i Yes CD No CD I don't know. 

4. Is there an encyclopedia in youv home? ^ 
a Y« a No cd I don't know, 

» ? 

5* Hew much school did your father complete? 

(FILL IN THE ONE OVAL which best shows how much school your 
father completed.) 
CD Dta not complete the 8th grade 
CD Completed the 8th grade, but did not go to'high school 
, CD Went to high school, but did not graduate from high school 
CD Graduatedtrom high school' 
CD Some education diet graduation from high school 

r-> I don't know. 

6. Did your father graduate from a college or university? 
CD Yes CD No CD I don't know. 

7. How much school did your mother complete? * 

(FILL IN THE ONE OVAL which best shows how much school your 

mother completed.) , 

CD Did not complete the 8th grade 

CD Completed the Sth grade; but did not go to high school 

CD Went to high school, but did not graduate from high school 

CD Graduated fror& high school 

CD Some education after graduation from high school 

CD Idbntknow. " ' 

8. Did your mother graduate from a college or university? 
CD Yes cd No CD I don't know. 

9. Where did you live on your ninth birthday? 

CD In the United States (Please specify the state or territory.) 



CD Outside the United States (Please specify the country.) 



CD I don't know. 
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Standard Background Information Form for 17-Year-Olds 



1. Which of the following does your family have at home? (Fill in one oval on 
each line.) 

/ ' Have D o not have 



A. 


Newspaper received regularly . 


CD 


CD 


B. 


Magazines received regularly 


CD 


CD 


C. 


More than 25 books 


CD 


CD 


D.. 


Encyclopedia 


CD 


CD 


E. 


Dictionary 


CD 


CD 


F. 


Record player 


CD 


CD 


G. 


Tape recorder or cassette player 


CD 


CD 


H. 


Typewriter 


CD 


CD 


I. 


Vacuum cleaner 


CD - 


CD 


J. 


Electric dishwasher 


CD 


CD 


K. 


Two or more cars or trucks that run 


CD 


CD 



2. How much time did you spend on homework yesterday? 



cd No homework was assigned « - * v - 

o I had homework but didn't do it . 

c=6 Less than one hour 

czd Between 1 and 2 hours 

cd More than 2 hours 

3. How many different schools have you attended si«oe you started the first 
f y grade? * 

cd 1 to 3 schools 1 % 
CD 4 to 6 schools 
cd 7 to 9 schools, 
cd 10 or more schools* 

V 

4. How long have you lived in the community in which you now live? 
cd All my life 

cd 10 or more years, but not all my life 
cd 5 to 9 years 

"~ C5~2 to 4 years * ■ 

cz> 1 year 3 ■ 

cd Less than 1 year 

5. How much television did you watch yesterday? 

cd Norte cd 2 hours cd '5 hours 

cd fchour or less cd 3 hours cd 6 hours or more 

cd 1 hour * C3> 4 hours 

67 Ts English the~lahguage spoken most often frryour home? 

cd Yes cd No 




7. Is a language other than English spoken in your home? 

a often , cd Sometimes o Never 

8. How many brothers or sisters do you have who are older than you? 

None 1 2 3 4 5 6 or more ; 

CD CD CD CD CD O CD 

9. How many brothers or sisters do you have who are younger than you? 

None 1 2 3 4 8 6 or more 

CD CD CD CD CD CD CD 

10. A. Wfiat is your racial background? 

cd American Indian or Alaskan Native 

cd Asian or Pacific. Islander ^ 

cd Black » 

cd White 

CD Other (Please specify) 



B. Is your ethnic heritage Hispanic (such as Mexican. Puerto Rican^ « 
Cuban. Central or South American or other Spanish Culture or origin)? 



> Yes cd No 



11. How often hgs each of the following been used in the courses you are taking 
this year? (Pill inone oval on each line.) 



Fairly i 
Never Seldom Often Frequently 



A. 


Xistening to the teacher's lecture 


CD 


CD 


CD 


CD 


B. 


Participating in student-centered 
discussions 


CD 


CD 


CD 


CD 


C. 


Working on a project or in a 
laboratory 


CD 


CD 


CD. 


CD 


D. 


Writing essays, themes, poetry, 
stories 


CD 


CD 


CD 


. CD <■ 


E. 


Going on field trips 


CD 




CD * 


CD 


F. 


Having individualized instruction 
(small groups cr one-t<H>ne with 
a teacher) i 


CD 


CD 


CD 


CD 


G. 


Using teaching machines or computer^ 
assisted instruction 


CD 


CD 


CD 


CD 


H. 


Watching television lectures 


CD 


CD 


CD 


CD 


I. 


Studying from textbooks 


CD 


CD 


CD 


CD 






CD 


CD 


cp 


3r- 


— btbrary-or-media-eenter-assignments — cd 




* 





A How much school did your father complete? 

(FILL IN THE ONE OVAL which best shows how much school your 
father completed.) 

cd Did not complete the 8th grade - 
o * Completed the 8th grade, but did not go to high school 
cp Went to high school, tut did not graduate from high school 
cb Graduated from high school 
cz) Some education after.graduation from high school 
h i 

cd I don't know. 
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B. Did your father graduate from a college or university? 
cz, Yes c^> No cd I dor\'t know; " 

C. How much school did your mother complete? 

(PILL IN THE ONE OVAL Which best shows how much school your 
mother completed.) 

o, Did not complete the 8th grade 

o Completed the 8th gnpie, but did not go to high school 

o Went to high school* but did not graduate from high school ' 

o Graduated from high school 

Some education after graduation from high school 

\ 

feD I don't know. 

D. Diayour mother graduate from a college or university? 
CD Yes CD No cd I don't know. 

E. Where did you live on your ninth birthday? 

cd In the United States (Please specify the state or territory.) 



* cd Outside of the United States (Please specify the country.) 

cd I don't knbw. 
F, Where did you live on your thirteenth birthday? 

cd In, the United States (Please specify the state or territory.) 

t 

cd Outside of the United States (Please specify the country.) 
o I don't know. 
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Background Information? Art Museum or Gallery Visits 
9-/ 13- and 17-Year-Qids 



The works ofartists are shown in art museums and art galleries. How often 
have you visited art museums or art galleries? 



Never 1 
One time 
About five times 
About ten times 
Fifteen or more times 
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Background Information: Kinds of Artwork Done Outside of School 

9-Year-Olds 



Outside of school, what kinds of art work do you do? Tell whether you 
do each of the following things. 

A. Outside of school, do you draw? 

! 

czd Yes 
<=>Ko 

czd I don't know. 

B. Outside of school, do you paint? 

czd Yes 

No » 

czd I don't know. . J 

C. Outside of school, do you make collages by cutting and pasting paper, 
cloth and scrap materials? 

czd Yes 

czd No t 
czd I don't know. 

D. Outside of school, do y * carve or "make models with wood, stone, 
clay, metal or plastic? ( t % 

czd Yes 
o No 

czd I don't know. 



\ 
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Background Information: Kinds of Artwork Done Outside of School 

13- and 17-Yrar-Olds 



Outside of school, what kinds of art work uo you do? Jell whether you do 
each of the following things outside of school. 







Yes 


No 


I don't know. 


A. 


Drawing 


cz> 


o 


cz> 


E. 


Painting 


cz> 


o 


o 


C. 


Making pictures by cutting and 
pasting paper, cloth and scrap 
materials 


\ 


o 


o 


D. 


Carving or modeling in wood, 
stone, clay, metal or plastic 


o 


a 


o 


E. 


Print-making such as block 
printing, silk screening, 
etching 


o 




o 


F, 


Making potteiy, ceramics or 
mosaics 


o 


o 




G. 


Weaving, macrame or knot- 
tying, or needlework such as 
embroidery, needlepoint, 
knitting, crocheting 


o 


o 


cz> 


Hi 


Making photographs or films 


cz> 


o 


cz> 


il- 


Making jewelry 


o 


o 


CD 


J. 


Creating designs or plans for 
, things like clothes, toys, cars, 
houses, furniture 


cp 


a 


) <=> 

i 



r 
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Backgrdbrri Information: Art Classes Taken — 13-Year-Olds 



A. Did you take an art class in school last year? 

cd Yes cd No • cd I don't know* 

B. Are you tak^ an art class during this school year? 
cd Yes cd No cd I don't know. 

. t 
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Background, Infotxnation: Art Classes Taken — 17-Year-01ds 



A. Please indicate in which of the following years you took art classes in 
school. 





Yes > 


No 


I don't know. 


Seventh grade 




CD 


CD' 


Eighth grade 

* 


CD 


] 

CD 


CD 


Ninth grade 


CO 


CD 


CD 


Tenth grade 






CD 


Eleventh grade 


CD 


CD 


CD 


Twelfth grade 


t 


CD 


CD 



B. Are you currently enrolled in an art class? 



Yes * cd No ^-CDrrdon't know. 
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Background Information* Types of Art Collected — 13-Year-Olds 



/ ' 

/ 

A. Original works of art are such things as paintings, drawings, sculpture, 
ceramic pieces, jewelry, hand-signed prints, weaving or any other work 
actually made by an artist. ^ 



Do you collect original works of art? 



Yes 
No 

\ I don't know. 



B. Reproductions of works of art are copies of original art works. 

t Do you collect reproductions of works of art? 

/ 

( oYes 
cz> No 

CD I don't know. 



ERIC 



52 



, 59 



Background InforraatdonrTyp^ of Art Collected — 17-Year-Oids 



A. Original works of art .are such things as paintings, drawings, sculpture, 
ceramic pieces, jewelry, hand-signed prints, weaving or any other work 
actually made by an artist/ 

Do you collect original works of art? . 

cz> Yes * ii 

cz> No " '< 

C3 I don't know. 
B- Reproductions 'of art works are copies of original art works. 

it i 
V 

' Do you collect reproductions of works of art? 

cz> Yes i 
p No „ 

cz> Idon!tknow. v 
C. Antique art old objects such as furniture, glassware, rugs and toys. 



Do you collect antiques-: 



cz> Yes 

c 

C3 No 



I don't know; 
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APPENDIX C 
RESPONSE RATES FOR ASSESSMENT SAMPLES 



Table " C-l shows the response 
rates for students. assessed in 
1974-75 and 1978-79. In. the 
1974-75 assessment, for each 
oi the three age groups, 12 
students and 4 altetnates were 
selected for each assessment 



session. If all 12 students 
appeared for the session, then 
the alternates were dismissed. 
Otherwise-, enough alternates 
were selected to bring the 
size- of the group up to or as 
near as pdssible to 12. If the 



TABLE C-l'.' Number of Students Assessed and Percent of 
Sample Covered, by Age and Assessment Year 

Average 
, Sample 
Covered 

in 
Percent 



Year 



1974-75 



1978-79 





Number 




of. 


Age 


Packages+ 


9 


6 (12) 


13 


7 (13) 


17++ 


7 (13) 


9 


11 


13 


' 13 


17- 


14. 



Total 
Number 
of 

Students 
Assessed 

28,403 
30,312 
29,612 

27,605 
#5,910 
^37,822* 



Average 
"Numbe r 
Assessed 

Per 
Package 

2,367 
2,332 
2,279 

2,510 
2,76^ 
2,702 



87.5 
83.7 
69.7 

87.8 
84.9 
77.8 



+ln the 1974-7-5 assessment, there were three reading 
booklets at each .age that were triple ' Wj^' JJJJ 
increases the effective number of booklets by six at- eacn 
aae The effective number of booklets is shown in 
parentheses and is used to determine the average number 
assessed per booklet, 

++At age 17, these numbers include both 17-yearrolds 
Enrolled and not enrolled in school for the reading and 
art assessments. In 1974-75, there were two *«t*onaX 
booklets of experimental materials and two booklets of 
materials administered under a special cojtr« c t- The 
respondents for these booklets are not included in these 
figures. 
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group assessed ^numbered be- 
tween" 8 and 12 students , then^ 
the* administration was consid- 
ered complete". If the final 
total was not at least a quo- 
rum off 8, a, Second and some- 
times a third make-up session 
was held. The percentage? in 
Table C-l are* based ou the 
numbers of students assessed , 
from the original groups of 12 
selected and do not reflect 
the use of alternates'. 

For the 1978-79 assessment, 
slightly different procedures 
were used. The number of stu- 
dents selected for e^ch admin- 
istrative session varied from 
16-25 students , depending on 
previous response rates ob- 
tained from schools in similar 
communities. No alternates 
were selected. The quorum size 
'needed to consider an adminis- 
trative session complete var- 
ied according to the number of 
students selected. Since non- 
response , rates have always 
been relatively small for, ages 
9 s and 13, , the make-up or 
follow-up procedures used in 
1978-79 for those ages were 
similar to the ones used for 
the first art, assessment. If a 
quorum was not obtained at the 
first administrative session/ 
a'second and sometimes a third 
make-up session was held. At 
age 17< in Jzhe 1978-79 assess- 
ment, ; follow-up procedures - 
were conducted on a school, 
rather than a session, basis. 
If a school had an overall 
response rate of less than 
75%, then, all nonrespondents 
in the school were contacted 
for one or two follow-up ses- 
sions. These follow-up proce- 
dures for 17-year-olds pro- 



vided sample coverage similar 
to that obtained at ages 9 and 
13. 

Since response rate? at age 17 
have always been somewhat^ 
lower than at the other two' 
ages, the Research Triangle 
Institute (RTI) , Raleigh, 
North Carolina, was asked to 
conduct ,a special study of 
nonrespondents' during " the 
1972-73 assessment of science 
and mathematics. The results 
(Kalsbeek et al., 1975; Rogers 
et al. , 1977) indicated that 
about 80% of the total nonre- 
spondent group did not appear 
at the assessment sessions 
because of conflicting school 
activities or illness. # The 
remaining 20% did not seem to 
he available. They attended 
school- inf requentlyr, if at all 
(for practical purposes,, they 
had dropped out), or they had 
moved out of the school atten- 
dance area. In either case, 
these students probably should 
not have been listed in the 
in-school papulation of 
el igibles. . 

Tables ^published , in previous 
National Assessment reports 
showing response rates for age 
17- generally contain percent- 
ages adjusted to account for 
those 17-year-olds listed, but 
not * attending" school. But, 
since National Assessment has 
not had the resources in re- 
cent assessment years to rep- 
licate the RTI study, the 20% 
figure used as a basis for 
adjusting these percentages 
may be outdated, and thus the 
percentages given in Table C-l 
have not been adjusted « It 
seems likely that despite ef- 
forts to update the lists of 
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eligibles, these lists still 
contain some percentage of 
students who have in effect 
left the schools. Thus, the 



percentages listed for age 17 
are probably underestimates of 
the actual response rates tor 
17-year-olds attending scl.^ol. 
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APPENDIX D 



COMPUTATION OP MEASURES OF ACHIEVEMENT, 
CHANGES LN ACHIEVEMENT AtiD STANDARD ERRORS 



Several measures of achievement that 
National Assessment uses in its re- 
ports are described in Chapter 7 of 
this document. The, sample design, as 
described in Chapter 4, is a complex, 
deeply stratified, multistage proba-* 
bility sample design. Measures of 
achievement are obtained by weighting 
individual responses .appropriately. 
Reasonably good approximation of 
standard error estimates of these 
achievement measyres can be obtained 
by applying the jackknife procedure 
to, first-stage sampling units within 
strata, using the method of sbcces- 
siVe differences and accumulating 
across strata. 

In this section, the measures of 
achievement are first defined in al- 
gebraic form, followed by a descrip- 
tion .of the jackknife method that 
National Assessment uses to estimate 
the standard m errors of achievement 
measures. 



Measures of Achievement 

Based on the sample design, a weight 
is assigned to every individual who 
responds to an exercise (administered 
in an assessment. The weight is the 
reciprocal of the probability of se- 
lecting a particular individual to 
take a particular exercise with ad- 
justment for nonresponse. Since the 
probabilities of selection are based 
on an estimated number of people in 



the target age population, the weight 
for an individual estimates the num- 
ber of similar people that individual 
represents in the age population. As 
explained in Appendix E, the weights* 
are adjusted " to reflect information 
from previous assessments on popula- 
tion distributions. 

A sum of the weights for all individ- 
uals' at an age level responding to an 
exercise is an estimate of the' total 
number of people in that age popula- 
tion. A sum of ^weights for all indi- 
viduals at an age responding cor- 
rectly to an exercise is an estimste 
of the number of people in the. age 
population who would be able to re- 
spond correctly if the entire popula- 
tion were assessed . These concepts 
also apply to any reporting group 
(e.g., defined by region, sex, and so 
on) and category of response (e.g., 
correct, incorrect and "I don't 
know") . 



Let W? u . 
ihk 



sum of weights for respon- 
dents to exercise e who 
are in reporting subgroup 
j/ ard who are in the kth 
replicate of the hth -sam- 
pling stratum, and 



C ihk ~ sum of w^hts for respon- 
dents to exercise e who 
are in reporting subgroup 
JL, who are in the kth rep- 
licate of the hth sampling 
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stratum and who selected 
• response category j_ ( e *9*r 
correct foil) for the ex- 
ercise. 



portion in subgroup i_ who would se- 
lect category 2 on exercise e and the 
proportion in the nation is denoted 
by: 



Note that wf hR = ? 



Then summiftg k over the n^ sample 

replicates in the stratum h, and sum- 
ming over the H sampling strata , 

W? +f = Z E h Wf hR . 
h=l k=*l 

estimates the number of eligibles in 
the population who are in subgroup i_. 



Similarly, C. 



1++ h=l k=l > 



mates the number of eligibles in the 
population who are in subgpup i_ and 
who woulc^ select response category 2 
for exercise e. * 

An estimate of the proportion of the 
eligibles in the age population in 
group i. who would select response 
category 2 on exercise £ is: 



(3) AP* j - - P *i. 
1 i A 



National Assessment also reports the 
arithmetic mean of the percentage of 
correct responses over sets of exer- 
cises corresponding to the measures 
in (1) , (2) arid (3) . < These means are 
taken over the set of all exercises 
or a subset of exercises classified 
by 7 a reporting topic or consent ob- 
jective. The meaft percentages of 
correct responses taken over m\xer- 
cises in some set of exercises corre- 
sponding to measures (1) , (2) and \S) 
are, respectively: \ 



(A) P< - 



e 



(6) AP i + P jL 



(1) Pj J - 



C ej /W e 



In the special case where the pro- 
portion ,of all age-el igibl^s who 
would select response category 2 on 
exercise e is estimated; the index A 
(for ALL) will be used in place of i. 
as follows: 



(2) - 

In National Assessment reports, the 
proportion in (1) multiplied by 100 
is called the group percentage, and 
the proportion in (2) multiplied by 
100 is called the national percent- 
age. The difference between the pro- 



Note that the response category sub- 
script 2 has been suppressed since 
the means are, understood to be taken 
oven the correct response category 
for each exercise. 

Each of these six achievement meas- 
ures is computed and routinely used 
in reports describing achievement 
data for any assessment. The simple 
difference in these measures between 
two assessments of the same exercise 
(or sets of exercises) provides six 
measures of change in achievement 
that are routinely used in National 
Assessments change reports. The next 
section describes how standard errors 
are estimated for the 12 stati/stics 
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used in NAEP reports • 

Computation of 
Standard Errors 

In order to obtain an approximate 
measure of the sampling variability 
in the statistics (1) through (6) , a 
jackknife replication procedure for 
estimating the sampling variance of 
nonlinear statistics from complex, 
multistage samples was tailored to 
National Assessment's sample design. 
Miller (1968, 1974) and Mosteller and 
Tuke? (1977) provide information 
about the jackknife technique, while 
Folsom (1977) describes how the pro-^ 
cedure is used in estimating standard 
errors for National Assessment's sam- 
ple design. 

Tto - demonstrate the computational H as- 
pectS of this technique, consider 
estimating the variance of the sta- 
tistic in .(1) — the proportion of 
age-el igibles in subgroup who would 
select "response categpry 2 on exer ~ 
cise £. 

This statistic is based on the data 
from all "the n^ replicates , in the H 

strata. Let P?£ hk be defined as a 

replication estimate of P^ and 

constructed from all the replicates 
excluding the data from replicate k 
in stratum h* These replication esti- 
mates are computed as if the excluded 
replicate had not responded, and a 
reasonable nonresponse adjustment is 
used to replace the .data' in replicate 
hk in estimating P? 3 . Several choices 

for replacing the data in replicate 
hk are available. In order to obtain 
a convenient and computationally ef- 
ficient algorithm for approximate 
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ing standard errors,. National Assess- 
ment replaces' C?^ k and W? hk from the 
hkth replicate with corresponding 

sums from another paired replicate in 
the same stratum. The replicate esti- 
mate is -then, computed. The replicate 
estimates to. be used in the calcula- 
tions are determined by arranging all 
the replicates in each stratum into 
successive pairs. That is, replicate 
1 is paired with replicate 2,. repli- 
cate 2 with replicate 3, 3 with 
4,...(n h -D with and repUcate in^ 

with replicate 1. 



The contribution to the variance of 

P e 3 by each pair of replicates is the 

change in the value of the statistic 
incurred by replacing the data from 
each replicate in the pair with the 
data from the other replicate in the 
pair . arri recomputing P^ in the 

usual way. This produces two repli- 
cate estimates. Squaring the differ- 
ence between these replicate esti- 
mates and then dividing by eight 

measures the contribution of this 
pair of replicates to the total vari- 
ance. The sum of these contributions 
over all n^ successive pairs in the 

stratum is the contribution by stra- 
tum hi to the total variance . The * 
square root of the sum of the H stra- 
tum contributions is th|. estimate of 
the standard error of P. . 
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Algebraically, the two* replicate es- 
timates for the pair k, k+1 (where 
kpl,...^ and are: 



(7) P 



C i-H- C ihk * U ih(krfl) 



i-hk 



and 



(8) P i-h(kfl) 



rf u+ " w ihk *ih(k*l) 



c e J - c ej + c ej , 

C i+f. C ih(kf l) ihk 
W i+f W ih(kfl) W ihk 



The contribution to the total vari- 
ance from stratum h is: 



where P=*P^, Q=l-P and N is the num- 
ber of. sampled respondents in sub- 
group ^ who took the exercise. The 
larger size of SE(P^) reflects 

mainly the loss of precision due to 

cluster-sampling of schools and stu- 
dents. The standard errors for the 
achievement measures (2) through (6) 
are computed through a series * of 
steps analogous to, those followed in 



(9) var (P*j[) 



?Vp e3 * 

I ( ? i-hk 



ej \2 ^ computing SE(Pp) . 



i-h{kH) / 



And finally, an estimate of the stan- 
dard error of P e] is; 

i 



(10) SE (P* j ) 



(E var P^) . 



Multiplying pJJ by 100 H yields the 

percentage of response to category 2» 

Multiplying S^(P^) by 100 yields 

the corresponding estimated standard 
error of the percentage. 



In general , the jackknifed standard 
errors of the^ proportion estimates 
will be larger than the simple random 
sampling formula (PQ/N) 1/ 1 9 



The standard * errors for the differ- 
ences between two assessments for any 
of the achievement measures (1) 
through (6) are computed as the 
square root of the sum of the squared 
standard errors from each of the sep- 
arate assessments. 

The size 6f the standard errors de- 
pends largely not 'only on the number 
of replicates and schools included in 
the sample , but also on the number of 
respondents in each of the reporting 
groups. Table I>1 shows the average 
number of students responding to an 
exercise booklet for each of the re- 
portinq qroups for each age for each 
assessment year. Table D-2 shows 
National Assessment's current esti- 
mates of the proportions of students 
an each reporting group at each, age. 
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TABLE EKU Average Number of- Respondents in Reporting Groups 
Taking an Item Booklet, by Age and Assessment Year* 



Age 1 9 
1974-75 1978-79 



Age 13 
1974-75 1978-79 



Age 17# 
i974-75 1978-79 



Nation 



2,382 



2,510 



,2,388 



2,762 



2,168 



2^,702 



Region 
Northeast 
Southeast 
Central 
West 



581 
564 
695 
542 



580 
625 
665 
639' 



601 
559 
686 
542 



675 
657 
752 
-681 



536 
501 
634 
.496 



642 
683 
725 
649 



Sex 
Male 
Female 



1,197 
1,184 , 



1,255 
1,255 



1,172 
1,216 



1,370 
1,395 



1,050 
1,118 



1,312 
1,386 



Race/ethnicity 
White 
Black 
OtherH- 



1,844 
396 
142 



1,849 
484 
177 



1,884 
374 
130 



2,053 
507 
202 



1,794 
279 
96 



2,134 
392 
176 



Parental e du c at io n 

Not graduated high school 215 191 

Graduated high school 514 589 

Post high school . 758 880 

. Unknown++ ' 895 850 



355 
746 
973 
314 



326 
895 
1,200 
341 



357 
696 
993 
122 



386 
895 
1,334 
87 



Type of ccnmmity 
Rural 

. Disadvantaged urban 
Advantaged urban 
Other+f 



249 
236 
238 
1,659 



252 
250 
253 
1,755 



262 
227 
236 
1,663 



271 
281 
277 
1,933 



218 
228 
202 
1,521 



246 
306 
265 
1,885 



Size of camrnunity 

Big cities 460 723 

Fringes around big cities 450 448 

Medium cities 346 237 

■Small places 1,1?6 1,097 



Grade 

3, 7, 10 

4, 8, 11 
12 

Other++ 



522 
1,793 

67 



623 
1,834 

53 



460 
472 
326 
1,131 



615 
1,713 

60 



775 
588 
287 
1,115 



683 
2,031 

48 



390 
423 
326 
1/030 



282 
1,593 
253 
36 



718 
571 
287 
1,123 



360 
2,014 
285 
43 
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+Data may not total toue to binding error. 
#Seventeen-year-olds 'enrolled in school. 
++Data are not reported for these groups. 
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••• TABLE D-2. Estimated Current 
qf National Assessment 
for In-School 

Reporting Groups 

Sex 
Male 
Female 

Race/ethnicity. 

White 
„Black 

Other 

Region 
Northeast 
Southeast 
Central 
West 

Parental education 
Not graduated high school 
Graduated high school 
Post high school 
Unknown 

Type of community 
Rural 

Disadvantaged urban 
;ivantaged urban 
Other 

Size of community 
Big cities 

Fringes around big cities 
Medium cities 
Small places 

Grade .in school 

<3, <7 , <10 

3, 7, 10 

4, 8, 11 
>4, >8, 12 
Other 



Population Proportions 
Reporting Groups 
Students 



e 9 Age 13 Age 17 



.50 .50 .48 

..50 .50 .52 



.7? .80 .83 

.i4 .13 .12 

.07 .07 .05 



.25 .25 .25 

.22 .23 .20 

.27 .27 .29 

.26 .25 1 .26 



.09 .13 .15 

.24 - .32 .32 

.33 .42 .48 

.34 .13 . .05 



.08 .10 .08 

.07 .07 .09 

.11 .11 .11 

.74 .72 .72 



.20 .21 .19 

.22 .22 .26 

.12 .11, -11 

.46' .46 .44 



< c 01 .02 .02 

.23 .25 .13 

.75 .72 .75 

<.01 <.01 ^ .10 

<.01 <.01 <.01- 
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APPENDIX E 

ADJUSTMENT OF RESPONDENT WEIGHTS BY SMQOTHING TO REDUCE 
RANDOM VARIABILITY OF ESTIMATED POPULATION PROPORTIONS 



Background 

A weight is assigned to every 
individual who responds to an 
exercise administered in an 
assessment. The weight is the 
reciprocal of the probability 
of selection of the individual 
with adjustment for nonre- 
spbnse. The weight for an 
individual estimates the num- 
ber of people that the indi- 
vidual represents in ' the age 
population. The sum of the 
weights of all individuals at 
an age level who responded to 
an exercise is an .estimate of 
the total number of people in 
that age population in the 
year that the exercise was 
assessed. Similarly, the sum 
of weights for all Individuals 
who took the exercise and who 
also are members of some demo- 
gr*otiic category (such as 
blc ~ks) gives an estimate of 
the number of people in the 
age population, f c ; the ysar, 
^ho are also members of the 
category.' The ratio of the two 
totals estimates the propor- 
tional representation , of the 
demographic category in the 
age population for the given 
year. 

Separate estimates of the pro- 
portional representation of 
the various demographic sub- 
groups are provided by each 



booklet administered to a 
particular age group in a 
given year. Because of sam- 
pling variability, the esti- 
mates of population .propor- 
tions for a given year based 
on single booklets" vary. 
There is also sampling varia- 
tion in estimates of popula- 
tion proportions from year to 
year in addition to any exist- 
ing trends in population pro- 
portions over time. 

It is desirable to reduce the 
random variability of popula- 
tion proportions as much as 
possible, since this variabil- 
ity has an effect on perform- 
ance estimates . For example, 
. the percentage of acceptable 
responses for- an age group is 
a function of the relative 
proportions of high-performing 
and low-performing groups. If 
the relative proportions of 
these groups are very differ- 
ent in different assessments 
due to sampling variability, 
then a portion of the change 
in percentage of acceptable 
responses for an age group 
could be attributable to 
yearly sampling differences in 
the relative proportions- of 
high- / and low-achieving 
groups. 

In addition to reporting per- 
formance estimates for an age 
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.group as a whole, National 
Assessment also reports per- 
formance for various subpopu- 
lations, such as whites, or 
blacks. Because variability of 
subgroups within these sub- 
populations (such as males and 
females within the white popu- 
lation)' iafluences the per- 
formance estimates for the 
subpopulations, it is desira- 
ble chat fluctuations of pro- 
portions of all subgroups of 
each subpopulation* be reduced 
as much as possible. 

For e£ch age and year, each of 
the various booklets adminis- 
tered 'provides estimates of a 
given population proportion. 
Since these estimates are sub- 
ject to booklet-to-booklet 
variability, a better estimate 
of the population proportion, 
which will have reduced varia- 
bility, is obtained by combin- 
ing the information from all 
booklets. However, these pro- 
portions vary from y«^ar to 
year due to random sampling 
variability or systematic dTF 2 ^^ 
ferences in sampling proce- 
dures. An even better estimateof 
population proportions for any 
single year can be obtained by 
smoothing the proportions over 
several assessment years. The 
word "smoothing" is u^-ed' here 
in the sense of fitting a 
smooth curve to a sequence of 
numbers by robust/ resistant 
procedures (Tukey, 1977) . 
Smdothing estimates of popula- 
tion proportions reduces * a 
'large portion of the sampling 
variability while preserving, 
as far as* possible, actual 
trends occurring in the age 
population. 

After the population propor- 



tions f have been smoothed, ad- 
justed' weights, are derived for 
} the assessed individuals so 
that the population propor- 
tions computed using the ad- 
justed weights are equal to 
the smoothed proportions . The 
ad justed. weights are then used 
for all analyses. 



Smoothing Procedures 
Used by National 
Assessment 

The most direct way to smooth 
proportions i§ first to clas- 
sify people into mutually ex- 
clusive multiway cells on the 
basis of their membership in 
categories of various impor- 
tant* variables and then to 
smooth * the proportions within 
each of the resulting multiway 
cells across years. Unfortu- 
nately, this procedure tends^ 
to produce a large number of 
cells with few people and, 
consequently, quite unstable 
estimates of smoothed propor- 
tions. 

To circumvent this difficulty, 
National Assessment has uti- 
lized various smoothing proce- 
dures. These procedures, w^ \ch 
arte all basically weighting- 
class adjustments appl ied 
independently to each age, are 
designed to control, to var- 
ying degrees, fluctuations in 
certa in key subgroups wh ile 
avoiding, as m'uch as possible, 
instabilities due to small 
cells . 

/ 

The procedure used in 1978-79 
has the following characteris- 
tics: 

1. It produces a single ad 
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justed weight 
individual • 



for 



each 



2. Ib affords good control on 
the distribution of pro- 
portions of certain key 
variables . 

3. It tends to produce sta- 
bility of performance es- 
timates. 

4. It is relatively easy to 
~~ implement. 

Even though adjusted weights 
using ^Ys procedure differ 
slighfly^ from the correspond- 
ing adjusted weights from the 
other procedures- that have 
previously been employed, Na- 
tional Assessment intends to 
use weights obtained using the 
1978-79 procedure for all fu- 
ture analyses of data assessed 
in earlier years. This issimr 
ply because we believe weights 
obtained through this proce- 
dure have the most desirable 
features. 



The Current Smoothing 
Procedure 

The first step in the 1978-79 
smoothing procedure involved 



J 

the*, partitioning of the popu- 
lation of age-class eligible© 
into ' the six smoothing cells 
given in -Table E-l. The same 
cells were used for all ages. 

Then* for each age and every 
year , the proportion of the 
population in 'Bach of the 
cells was estimated. For a 
given age and year, the pro- 
portion of the population in a 
particular cell was computed 
as the sum of .weights of all 
respondents assessed in the 
given year who we re of the 
specified age and* who belonged 
in ,the cell/ divided : by the 
total of the weight of all 
respondents of the given age 
assessed in that year, .f* 



Each of the six cells was com- 
prised of a sequence of esti- 
mated population proportions 
corresponding to the various 
years of assessment, * Each 
such Sequence of proportions 
was then smoothed by fitting 
robust/resistant lines . Using 
data from the U.S. Census Bu- 
reau and Current Population 
Survey, trends in enrollment 
by age and race and by age and 
region were obtained* The dat? 
from these surveys were ad- 



TABLE E-l« Smoothing Cells Used for the 
1978-79 Smoothing Procedure 



Cell 


Race 


Region 


Community Size 


1 


White 


All 


Big cities + fringe 


2 


White 


All 


Medium cities 


3 


White 


All 


Small places 


4 


Black. 


Southeast 


All 


5 


Black 


Not Southeast 


All 


6 


Other 


All 


All 
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justed to correspond with Na- 
tional Assessment definitions 
as much as » possibly. The re- 
sistant lines wixhin the 
smoothing cells were con- 
strained to satisfy the trend 
from the U.S. Census and Cur- 
rent Population Survey data. 

The final step in- the smooth- 
ing procedure was ;to adjust 
the respondents' weights to be 
consistent with the smoothed 
proportions. Since each re- 
spondent takes only one book- 
let, the weight adjustments 
were done independently for 
each booklet. For a given age, 
year and booklet, population 
proportions using the original 
weights were obtained for each 



of the smoothing cells. Then 
the, weights' of all respondents 
of a given cell were multi- 
plied by the ratio of the 
smoothed cell proportion to 
the proportion using the, orig- 
inal weights. This produced 
the adjusted weights that we're 
used in all analyses. 



Adjustment of Weights 
by Users 

The smoothed population pro- 
portions for 9- r 13- and 17- 
year-olds (in-school only) are 
given in Tables E-2, E-3 and 
E-4, respectively. The columns 
of each table represent the 
smoothing cells while the rows 



TABLE E.-2. Smoothed Frequencies From 10-Year Smooth 
by Smoothing Cell and Year for 9-Year-01ds 



1 

White 
All 



Race 
Region 
Size of 
Community BC+FR# 



2 

White 
All 

MC++ 



Cell 



3 

White 
All 

SPH 



4 

Black 
GE+ 

All 



Black 
Not SE 

All 



6 

Other 
All 

All 



Year 

1970- 71 

1971- 72 

1972- 73 

1973- 74 

1974- 75 

1975- 76. 

1976- 77 

1977- 78 

1978- 79 

1979- 80 



0.3299 
0.3232 
0.3165 
0.3098 
0.3030 
0.2963 
0.2896 
0.2829 
0.2762 
0.2694 



0.1203 
0.1177 
0.1152 
0.1126 
0.1101 
0.1076 
0.1050 
0.1025 
0.1000 
0.0974 



0.3574 
0.3d47 
0.3720 
0. 3793 
0.3866 
0.3938 
0.4011 
0. 4084 
0.4157 
0.4230 



0.-0557 
0.0562 
0.0568 
0.0573 
0.0579 
0.05S4 
0.0590 
0.0596 
0.0601 
0.0607 



0.0736 
0.0743 
0.0749 
0.0756 
0.0763 
0.0770 
0.0776 
0.0783 
0.0790 
0.0797 



0.0631 
0.0638 
0.0646 
0.0654 
0.0661 
0.0668 
0.0676 
0.0684 
0.0691 
0.0698 



+SE - Southeast. 

#BC+FR = big cities + fringes. 

++MC « medium cities. 

##SP = small places. 
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TABLE E-3. Smoothed Frequencies From 10-Year Smooth 
by Smoothing Cell and Year for ,13-Year-Olds 

6 

Other - 
All 

All 

0.061Q 
0.0618 
,0. 0626 
0.0634 
0.0642 
0.0650 
0.0658 
0. 0666 
0.0674 
0.0682 





1 


2 


3 


4 


5 


Race 


White 


White 


White 


Black 


Black 


Region 


All 


All 


All 


SE+ 


Not SE 


Size of 










All 


Community 


BC+FR# 


'MC++ 


SP## 


All 


Year 










- 

0.0679 


1970-71 


0. 3327 


0.1113 


0.3748 


0.0523 


1971-72 


0.3279 


0.1106- 


0. 3779 


0.0524 


- 0.0694 


1972-73 


0.3232 


0.1098 


0.3180 


0. 0525 


0.0709 


1973-74 • 


0.3184 


0.1091 


0.3841 


0. 0526 


0.0724 


1974-75 


0.3137 


0.1084 • 


0.3872 


0.0527 


0.0739 


1975-76 


0.3089 


0.1076 


0.3903 


0. 0528 


0.0754 


1976-77 


0.3042 


0.1069 


0. 3933 


0.0528 


0.0770 


1977-78 


0.2994 


' 0.1062 


0.3964 


0.0529 


0. 0785 


1978-79 


0. 2946 


0'.1055 


0.3995 


0.0530 


0. 0800 


1979-80 


0. 2899 


0.1047 


6.4026 


o;o53i 


0. 0815 



+SE = Southeast. 

#BC + FR - big cities + fringes 

++MC = medium cities. 

##SP - small places. 



TABLE E-4. Smoothed Frequencies From 10-Year Smooth 
by Smoothing Cell and Year for In-School 17-Year-Olds 



Year 

1970- 71 

1971- 72 

1972- 73 

1973- 74 

1974- 75 

1975- 76 

1976- 77 

1977- 78 

1978- 79 

1979- 80 



0.3634 
0.3577 
0.3519 
0.3462 
0.3404 
0.3347 
0.3290 
0.3232 
0.3175 
0..3117 



2 

White 
All 

MC++ 



3 

White 
All 

SP## 



4 

Black 
SE+ 

All 

0. 0438 
0. 0444 
0.0451 
0.0457 
0.0463 
0.0470 
0. 0476 
0. 0482 
0. 0489 
0.0495 



5 

Black 
Not SE 

All 

0.0581 
0.0597 
0.0614 
0.0630 
0.0647 
0.0663 
0.0679 
0.0696 
0.0712 
0.0729 



6 

Other 
All > 

All 

0.0472, 

0.0478 

0.0484 

0.0491 

0.0497 

0.0503 

0. 0509 

0.0515 

0.0522 

0. 0528 



1 

Race White 
Regicn All 
Size of 
Community BC+FR# 



0M205 0.3670 

0.1199 0.3704 

0.1194 " 0.3738 

0.118° 0.3772 

0.118J 0.3806 

0.1177 0.3,840 

0.1172' 0.3874 

0.1166 0.3907 

0.1161 0.3941 

0.1155 0.3975 



+SE = Southeast. 

#BC+FR = big cities + fringes. 

++tyC = medium cities. 

##SP = small places. 
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represent the assessment year* 
For example , the smoothed pop- 
ulation proportion of 9-year- 
olds in smoothing cell 2 
(whites in 'medium cities) for 
1952-73 is .1152. 

To adjust respondent weights 
to be consistent with the 
smoothed proportions, the fol- 
lowing procedures were fol- 
lowed : 

1 # For each booklet, respon- 
dents , were classified ac- 
cording to smoothing cell, 
and the raw population 
proportions for each cell 
were obtained. For exam- 
ple, the raw proportions 
for a booklet given to 
9-year-olds in smoothing 
cell 4 was the total of 
the' weights of all 9- 
year-olds receiving the 
booklet who were black and 
in the Southeastern re- 
gion, divided by the total 
of the weights of all re- 
spondents .receiving ' the 
booklet. 

2. For each booklet and 
smoothing cell, a weight 
adjustment factor as the 
ratio of the' smoothed pop- 
ulation proportion (for 
the appropriate age, year 
a nd smoo th i ng cell) ove r 
the raw population pro- 
portion was obtained. 

3. The adjusted weights for 
an individual were the 
product of that individ- 



ual's original weight and 
the appropriate adjustment 
factor. 



Changes in Smoothed 
Proportions as New 
Assessments Are Completed 

to 

Every time an assessment is 
completed, a new time point is 
added to each of the sequences 
of population proportions 
within the smoothing cells.. 
This means that, even though 
robust/resistant procedures 
are used, the addition of a 
new point may somewhat change 
th3 values of smoothed propor- 
tions for prior years. Addi- 
tionally, any changes in meth- 
odology will* impact the esti- 
mates. 

This means that the smoothed 
proportions, with the addition 
of the next assessment data, 
are apt to differ somewhat 
from the corresponding 
smoothed proportions without^ 
the new .data. National As- 
sessment has adopted the 
philosophy that the smoothed 
proportions, based on all cur- 
rently available data - using 
the best available algorithm, 
are the best available. There- 
fore, all subsequent analyses, 
for any year; , will be done 
using this best available in- 
* formation-, even though this 
may produce estimates that 
differ slightly from * prior 
values. 
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GLOSSARY OP NATIONAL 



Acceptable response . Any re- 
sponse to an exercise that 
demonstrates achievement of 
the objective measured by 
that exercise. 

Administration time . The total 
time allowed on the paced 
audio tape for an exercise. 
(Includes the time allowed 
'for the stimulus and the 
response.) 

Administration timetable . Time 
periods during the school 

- year when the various age 
groups are assessed. The 
time periods are: 

Octobcr^Deconber - 13-year-olds 
January-February 9-year-olds 
March-May 17-year-olds 

Age group or age level . Three 
age groups nave 5een sam- 
pled in both art assess- 
ments: 9-yfe J ar>-olds, 13- o 
year-olds and 17-year-olds 
attending school.' Birth 
date ranges for each age 
group in each of the two 
assessments are as follows: 

Assessment Age 9 Age 13 Age 17 
1974-75 1965 1961 10/57-9/58 
1978-79 v 1969 1965 10/61-9/62 



Assessment . The documentation 
of the progress in knowl- 
edge, skills and attitudes 
of American youth* Measures 
are taken at periodic in- 



ASSESSMENT TERMS 

tervals for each learning 
area, with the goal of de- 
termining trends and re- 
porting the findings to the 
public and to the education 
community. 

Assessment administrator > In- 
dividual * employed to 
administer the assessment 
in participating schools. 

Background questions . Ques- 
tions about respondents' 
instructional experiences 
with art both in school and 
out of school were included 
in art item booklets. Stan- 
dard background questions 
asked in every learning 
area are found on the back 
pages of the item booklets 
and include such things as 
level of parental education 
and read i ng mu te r ia Is in 
the home. Background ques- 
tions used in the 1978-79 
art assessment appear in 
Appendix B. 

Booklet . Items (exercises) are 
presented to respondents in 
booklets. Booklets are de- 
signed to be scored by op- 
tical scanning machines. 
Each ' booklet contains: (1) 
instructions for answering 
items and sample items/ (2) 
assessment items and (3) 
background questions. Each 
booklet contains approxi- 
mately 30-35 minutes of 
assessment items and 10-15 
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minutes of introductory 
material and background 
questions* 

Category (scoring) . A classi- 
fication of a response to 
an open-ended exercise. See 
scoring guide. 

Category within a variable , A 
subclassif icatjion within a 
variable. For exatnple, male 
and female are categories 
of the variable sex. See 
reporting groups . 

Difficulty level : The percent- 
age of acceptable responses 
to an exercise. 

Exercise . A task designed to 
measure an objective* Be- 
cause NAEP does not admin- 
is te r "tests ," but instead 
desct ibes educational 
achievement over time, the 
term "exercise" is often 
used instead of the term 
"item" or "test item." The 
terms "item" and "exercise" 
are used synonymously in 
this report. 

Exercise booklet . See booklet . 

Exercise part . See item part . 

Exercise pool . The entire set 
of exercises prepared for a 
learning area. This set 
includes recycled exer- 
cises, exercises developed 
for previous assessments 
but not used due to exer- 
cise booklet or ^budgetary 
constraints and newly de- 
veloped exercises . 

Field test . A pretest of exer- 
cises to obtain information 
regarding clarity, diffi- 



culty levels, timing, fea- 
sibility and special admin- 
istrative problems. Needed 
£o\ revision and selection 
of exercises to be used in 
the assessment. 

Grade in school . Results are 
reported for 9-year-olds in 
grades 3 and 4; 13-year- 
olds in grades 7 and 8; and 
17-year-olds in grades 10, 
11 and 12. 

Group administration . Booklets 
were administered to groups 
of 10 to 25 students in 
1978-79. In 1974-75, normal 
group size was 12 respon- 
dents. A paced audio tape 
was used to provide uniform 
instructions and oral pre- 
sentation of exercises . 

Hand scoring (scoring) . The 
coding of responses in a 
format compatible with the 
optical scanning equipment 
being used. Multiple- 
choice exercises can be 
directly machine scored ; 
however, responses tp 
open-ended exercises must 
be coded in scoring ovals 
so that they can then be 
machine scored. See scoring 
guide , 

ID number . An identification 
number referring to the 
unique number assigned to 
each respondent. This num- 
ber is assigned to preserve 
the anonymity of each re- 
spondent. NAEP does not 
keep records of the names 
of any individuals. 

Item . See exercise . 

Item booklet. See booklet. 
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Item part * Each part of an 
item that asks a separate 
question. Parts may all 
pertain to one stimulus , 
such a graph or a table, 
or m-*/ concern the same 
topic* 

jackknife . The name of the 
algorithm used by NAEP to 
estimate standard errors of 
percentages and other sta- 
tistics. 

Learning area . One of the 
areas assessed by National 
Assessment: . reading/litera- 
ture, writing, mathematics, 
science, citizenship/social 
studies, art, music, career 
development. Also called 
"subject area." 

Le ^el- of parental education . 
These levels are described 
in Appendix A. 

Modal grade . The grade in 
which the majority of each 
in-school age group is en- 
rolled. For. ^year-olds, 
the modal grade (is grade 4; 
for 13-year-olds, grade 8; 
and for 17-year-olds, grade 
11. 

* 

Objective . A desirable educa- 
tion goal agreed upon by 
scholars in the field, edu- 
cators and concerned lay 
persons, and. established 
through the consensus ap- 
proach. 



Objectives 
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After the initial assess- 
ment of a learning area, 
one of the first steps in 
preparing for reassessment 
is a review of the 
learning-area objectives. 



This is carried out by 
scholars in the field, edu- 
cators and concerned lay 
persons. These reviews may 
result in revision, modifi- 
cation or total rewriting 
of the learning-area objec- 
tives to reflect current 
curricular goals and empha- 
ses; they may also result 
in the endorsement of the 
objectives from the previ- 
ous assessment as adequate 
for the next assessment. 

Open-ended exercise . A nonmul- 
tiple-choice exercise that 
requires some type of writ- 
ten or oral response. 

Paced audio tape . A tape re- 
cording that accompanies 
each booklet to assure uni- 
formity in administration. 
Instructions and exercises 
are played back from the. 
tape recording so that 
reading difficulties will 
not interfere with an indi- 
vidual's ability to re- 
spond. Response time is 
included on the tape. 

Primary sampling unit (PSU) . 
First-stage sampling, units, 
typically a county or a 
group of contiguous coun- 
ties. 

Principal ' s questionnai re . A 



data collect 
to school o 
officials re 
tions conce 
ments, size 
nity, occupat 
tion of tlje 
so forth. Sa 
questionna ire 
Appendix B. 
plementary 



ion form given 
fficials. The 
spond to ques- 
rning enroll- 
of the commu- 
ional composi- 
community , and 
mples of these 
s are found in 
See also sup- 
principal^ s 
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questionnaire . 

PSU . See primary sampling 
unit ; 

Public-use data tapes . Com- 
puter tapes containing re- 
spondent-level exercise and 
background/demographic data 
and machine-readable docu- 
mentation. These, tapes are 
available for use by exter- 
nal researchers wishing to 
do secondary ^analyses of 
National Assessment data. 

Racial/ethnic category . For 
EEe art assessments , re- 
sults are reported for 
whites, blacks and His- 
panos. (Because of a small 
sample size, only average 
results for Hispanos can be 
reported by National As- 
sessment.) 

Receipt control . Procedures 
Implemented by scoring 
staff to check in and 
screen materials^ from the 
field. Information gained 
from receipt control P£3ce- 
dures is relayed ^to the 
administrative staff so 
that any errors may be cor- 
rected. 

Recycled exercises . The set of 
' exercises tfiat is kept se- 
cure from one assessment to 
the next that will be used 
to measure changes (growth, 
stability or decline) in 
performance for the learn- 
ing area. 

Region . One of four geographi- 
cal regions used in gather- 
ing and reporting data: 
Northeast, Southeast,- Cen- 
tral and West.' States in- 
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eluded in each region are 
shown in Appendix A. 

Released exercise . An exercise 
for which results and exer- 
cise text have been re- 
ported to the public. 

Released exercise set . A set 
of released exercises, in- 
cluding documentation and 
scoring guides, that can be 
purchased from .National 
Assessment. 

Reporting groups . Categories 
of variables for which Na- 
tional Assessment data are 
reported . Variable cate- 
gories are defined in Ap- 
pendix A. 

Rescore . If an open-ended ex- 
ercise . was scored under 
different conditions than 
presently held or if pas- 
sage of. time might affect 
scoring, responses from a 
previous assessment may be 
rescored at the same time 
that responses from a later 
"assessment are scored. 
Responses from an earlier 
assessment also may be held 
and not scored so that they 
can be scored, with respons- 
es from a later assessment. 

Respondent . A person who re- 
sponds to the exercises in 
an assessment booklet. 

Response options . Different 
alternatives to a mul- 
tiple-choice question that 
can be selected by the re- 
spondent. 

Review conference «, A con- 
ference held to review the 
objectives of a learning 
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area to ensure their 
acceptance as measures of 
the* objectives by scholars, 
educators and lay persons 
or to reviey exercises for 
racialr ethnic, social or 
regional bias. 

Sample , National Assessment 
does not assess an entire 
age population but rather 
selects a representative 

9 sample from the age group 
to answer assessment items. 
(See " Chapter 4 for "a 
description of National 
Assessment s am pi ing 

procedures.) 

Scoring guide . A guide for 
hand scoring an open-ended 
exercise that specifies 
descriptive or diagnostic 
categories by giving 
definitions and sample re- 
sponses. 

Scoring ovals . Scannable ovals 
printed beside multiple- 
choice options and printed 
at the bottom of the page 
for open-ended exercises 
(to be used in hand scor- 
ing) . When .ovals* are 
marked , they can be scored 
by machine and responses 
then can be recorded by 
computer. 

Sex . Results are reported for 
males and females. 

Size of community . Results are 
Reported for four size.-of- 
community categories: big 
cities, fringes around big 
cities, medium cities and 
small places. These cate- 
gories are defined in Ap- 
pendix A* 



SMSA , Standard Metropolitan 
Statistical Area. SMSAs are 
economic and s&eial units 
defined by the U.S. Bureau 
of the Census. 



Standard error . A measure of 
sampling variability for a 
statistic . Because of 
NAEP's complex sample d^-~ 
sign, standard errors are 
estimated by jackknif ing 
first-stage sample esti- 
mates. 

Stem . The portion of an exer- 
cise that states the prob- 
lem or asks the question. 

Stimulus , For art exercises, 
this will be an aural 
and/or a visual stimulus 
used as part of the stem. 

Subject area . See learning 
area . 

Subpopulation or subgroup . 
Groups within the national 
population,-, such as ma^es 
and females, for which data 
are reported . 

Supplementary principal 's 

questionnaire . A data col- 
lection form given to 
school officials. On this 
form, officials are askqd 
to respond to questions 
concerning course offer- 
ings , materials and staf- 
fing specific to the learn- 
ing area being assessed. A 
sample of this question- 
naire is found in Appendix B. 
See also principal's ques- 
tionnaire . 

Tapescript, . A script prepared 
for the announcer to use in 
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producing the paced tape*. 
It indicates exactly what 
is to be read or not read 
aloud to the students and 
indicates the amount ' of 
response time allowed for 
each exerc ise . See paced 
audio tape . 

Timing . Most NAEP exercises 
are administered with a 
paced audio tape to stan- 
dardize data collection 
conditions. The tape in- 
cludes the amount of time 
students are allowed to 
respond to each exercise. 

Type of community . Results are 
reported for throe type- 
of-community categories: 
disadvantaged urban, advan- 
taged urban and rural. 
Definitions of these cate- 
gories are found in Appen- 
dix A. 



User tape . See public-use data 
tape . 

Variable . A classification of 
respondents . Standard re- 
porting variables are: re- 
gion, sex, race/ethnicity , 
level of parental educa- 
tion r size of community, 
type of community and grade 
in school. 

Weight . A multiplicative fac- 
tor equal to the reciprocal 
of the probability of a 
respondent being selected 
for assessment with adjust- 
ment for nonresponse — an 
estimate of the number of 
persons in the population 
represented by a respondent 
in the sample. Theoreti- 
cally, the - sum of the 
weights for all respondents 
at an ags level is equal to 
the number of persons in 
the country at that age 
level. 
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